AVs, Fatal Exceptions 17066, 17052, etc., etc.

I'm having a hard time with Sql Server 2000 at the moment after over a year of smooth operation. I hope someone here can help as I'm completely lost.

Sql Server 2000 SP2 - 8.00.534
Win2k
About 1400 databases.

After a while, sometimes 24 hours sometimes 2 hours sql server terminates with errors like the following (in order of occurrence).

17066 :
SQL Server Assertion: File: <S:\sql\ntdbms\storeng\drs\include\record.inl>, line=1447
Failed Assertion = 'm_SizeRec > 0 && m_SizeRec <= MAXDATAROW'.

18052 :
Error: 3624, Severity: 20, State: 1.

17066 :
SQL Server Assertion: File: <S:\sql\ntdbms\storeng\drs\include\record.inl>, line=1447
Failed Assertion = 'm_SizeRec > 0 && m_SizeRec <= MAXDATAROW'.

I've searched this forum and looked here - http://support.microsoft.com/common/canned.aspx?R=d&H=Microsoft%20SQL%20Server%202000%20Post-SP2%20Hotfixes&LL=&Sz=kbSQLServ2000presp3fix&Fr=&DU=&SD=GN&LN=EN-US&CND=1&VR=&CAT=&VRL=&SG=&MaxResults=200

but cannot find a fix.

I can provide the latest *.dmp file if you need it.

Anyone have any ideas?

--
PadraicRE: Originally posted by Podge
Q1 About 1400 databases.

Q2 <S:\sql\ntdbms\storeng\drs\include\record.inl>, line=1447
Failed Assertion = 'm_SizeRec > 0 && m_SizeRec <= MAXDATAROW'.

Q3 Anyone have any ideas?
--
Padraic

A1 That is quite a few databases (1400 are they all quite large)?

A2 Are you running internally created XP_ stored procedures at the time, (what is the purpose of the record.inl?), if so what do they do?

A3 Not really, it is interesting though. Some MS info does exist re. some bugfixes related to Error: 3624: http://support.microsoft.com/default.aspx?scid=kb;EN-US;274266|||Thanks for the reply. I really appreciate any help I'm getting.

1. Most databases are 1mb - 40mb, 10 are above 40mb and 2 are above 100mb.

2. The databases contain only default sp's. All db's are interacting only with asp pages (which don't call sp's). Don't know what "XP_ stored procedures" are.

3. From that page "This problem was first corrected in SQL Server 2000 Service Pack 1." I have Sql Server 2000 SP2 (which should include the fix ?)|||RE: Thanks for the reply. I really appreciate any help I'm getting.

1. Most databases are 1mb - 40mb, 10 are above 40mb and 2 are above 100mb.

2. The databases contain only default sp's. All db's are interacting only with asp pages (which don't call sp's). Don't know what "XP_ stored procedures" are.

3. From that page "This problem was first corrected in SQL Server 2000 Service Pack 1." I have Sql Server 2000 SP2 (which should include the fix ?)

1. Are they all very active? Is your server ever maxed out with respect to any resources (CPU, IO, etc.); if so is it busy at the time of the incidents?

2. XP_ = extended stored procedures.
If you have not developed any internally, and no outside contractor has developed any for you, and you have not now or ever had a non MS record.inl file in use, i.e.(S:\sql\ntdbms\storeng\drs\include\record.inl) ; Then this is most likely a MS bug.
==> In that case you should call MS support (ask for a refund as this is a MS Sql Server bug). <==

3. They may have fixed some issues (but not all, say your particular issue, for example?), or the MS fix may have created other issues?|||1. Some are very active, some are not at all. Since the errors started occurring I restricted use to about 80 db's instead of all 1400. Server is under no stress cpu wise but memory use keeps going up even though SQL server is set (not dynamic controlling memory).

2. I will as soon as I can find a number.

3. I believe there are a few more fixes available, I'm going to install them now. Cumulative right?

I have at the moment 8.0.0.584 (sp2.01)

Is this the latest?
http://support.microsoft.com/default.aspx?scid=kb;EN-US;316426|||1. ...no stress cpu wise but memory use keeps going up even though SQL server is set (not dynamic controlling memory).
2. I will as soon as I can find a number.
3. I believe there are a few more fixes available, I'm going to install them now. Cumulative right? I have at the moment 8.0.0.584 (sp2.01) Is this the latest? http://support.microsoft.com/default.aspx?scid=kb;EN-US;316426

1. Memory use keeps going up? (You don't mean as in a memory leak?)

2. {One last thought before calling MS PSS; you haven't experimented with enabling any unusual trace flags?} If not, support is available at $245 per incident:

Support for Phone Numbers
Small or Home Business (800) 936-4900
Developer (800) 936-5800
IT Professional (800) 936-4900
Partner (resellers or consultants) (888) 456-5570
Original Equipment Manufacturer (800) 936-2197
System Builder (888) 456-5570
Certified Partner (888) 677-9444
TDD/TTY (800) 892-5234

3. My most up to date installs are at:
8.00.679 <==> http://support.microsoft.com/default.aspx?scid=kb;en-us;316333 (Hotfixes often take more work to install, and undo)|||I have seen this occur when an index gets hosed. Are you using any clustered indexes ? If so, drop and recreate them. If that does not work, drop and recreate other indexes. I know you probably have quite a few but this is a necessary step.

Yes, please post your sql dump.|||RE: I have seen this occur when an index gets hosed. Are you using any clustered indexes ? If so, drop and recreate them. If that does not work, drop and recreate other indexes. I know you probably have quite a few but this is a necessary step.

Yes, please post your sql dump.

S1 You might want to run dbcc checkdb to confirm / assess any corruption (and get some idea of how extensive it might be), also PSS might want to have you do some other things as well.

RE: [I have seen this occur when an index gets hosed. ]

Q1 Is "index gets hosed" a cause, or an effect (or is / was that not clear)?
Q2 Do you happen to know the significance of record.inl, it looks to me like the Failed Assertion is comming straight out of MS source code (since it wasn't written by / for Podge)?|||DBA - Good idea about running dbcc checkdb. This might reveal even more information about whether or not sp2 somehow "modified" sp1. One of the fixes for sp1 relates directly to dbcc checktable (which checkdb uses). One word of caution - Only run dbcc checkdb when the usage of resources is at their lowest - also do not run backups while you do this and understand that this uses tempdb and will grow (there are more caveats - check out bol to see all of them and some handy options).|||Lots of clustered indexes in every db. I'll try to drop and recreate them.
This will take a while.

This is the latest error which resulted in downtime. No other severe errors occurred.

Error: 0, Severity: 19, State: 0
SqlDumpExceptionHandler: Process 57 generated fatal exception c0000005 EXCEPTION_ACCESS_VIOLATION. SQL Server is terminating this process.

I installed the hotfixes (the ones you mentioned were the ones I meant to post). Sql is now 8.00.679. It still crashed :(

I have run DBCC CHECKDB for the three largest db's. Anyone know how I can run it for all db's via a script?

Thanks for your continued help.|||Latest dmp file - 148kb

http://64.66.186.41/dmp.html|||Ran checkdb on all databases using this code

declare @.Sql varchar(1000)
, @.DbName varchar(100)

declare DbName cursor fast_forward for
select name from master.dbo.sysdatabases order by name

open DbName

fetch DbName into @.DbName

while @.@.fetch_status = 0
begin
set @.Sql = 'dbcc checkdb(''' + @.DbName + ''') with no_infomsgs'
print @.Sql
--exec (@.Sql)
fetch DbName into @.DbName
end

deallocate DbName

No errors.|||Interesting.

Q1 Is that the first time you have seen the AV you mentioned:
Error: 0, Severity: 19, State: 0 SqlDumpExceptionHandler: Process 57 generated fatal exception c0000005 EXCEPTION_ACCESS_VIOLATION. SQL Server is terminating this process.

Q2 Have you gotten any feedback from PSS on your service request? Has MS indicated that this is possibly a known bug, etc.?

Q3 I think it is important to try to determine if the indexes that are becomming corrupt is a cause, or an effect; as well as the significance of the record.inl file. Again, I suspect the Failed Assertion came straight out of MS source code.

Q4 Assuming periodic data / index checking / maintenance, etc., have you previously detected / repaired occasional index, linkage / or any other corruption issues? If so, is there any noticible pattern to it?|||Getting the 17052 error described in this thread, followed by SQL agent failing to complete jobs. The run through all steps, but fail to terminate and notify operators. Status shows 'Performing Completion Actions' pretty much for ever. Stop the agent but it usually won't start again and have to reboot. MSDB did show corruption on one occasion, and I was hopeful that fixing that would solve the problem. Unfortunately not. I'm looking for ideas too. SQL 2000 8.00.608 .