I have a table that I want to act as a queue.
It has no indexes and no key. Just one column.
Basically I want a stored procedure that will pull / return the first record off the queue (table) and delete it. I'd rather not use MSMQ for this.
There will be about 10 users trying to do this at the same time and will be trying to pull of about 15 times every second.
How can I do this and ensure that no two requests pull off the same row?
Thanks,
Kevin
Hello,
This is info that I am still not certain about and I just need to make sure, my gut feeling is correct:
A.
When a procedure is triggered upon reception of a message in a queue, what happens when the procedure fails and rolls back?
1. Message is left on the Queue.
2. is the worker procedure triggered again for the same message by the queue?
3. I am hoping the Queue keeps on triggering workers until it is empty.
My scenario is that my queue reader procedure only reads one message at a time, thus I do not loop to receive many messages.
B.
For my scenario messages are independent and ordering does not matter.
Thus I want to ensure my Queue reader procedures execute simultaneously. Is reading the Top message in one reader somehow blocking the queue for any other reader procedures? I.e. if I have BEGIN TRANSACTION when reading messages of the Queue, is that effectively going prevent many reader procedures working simultaneously. Again, I want to ensure that Service broker is effectively spawning procedures that work simultaneously.
Thank you very much for the time,
Lubomir
I am trying to implement a very fast queue using SQL Server.The queue table will contain tens of millions of records.The problem I have is the more records completed, the the slower itgets. I don't want to remove data from the queue because I use thesame table to store results. The queue handles concurrent requests.The status field will contain the following values:0 = Waiting1 = Started2 = FinishedAny help would be greatly appreciated.Here is a simplified script to demonstrate what has been done.CREATE TABLE [dbo].[Queue] ([ID] [int] IDENTITY (1, 1) NOT NULL ,[JobID] [int] NOT NULL ,[Status] [tinyint] NOT NULL) ON [PRIMARY]GOCREATE INDEX [Status] ON [dbo].[Queue]([Status]) ON [PRIMARY]GOCREATE PROCEDURE dbo.NextItem@JobID integer,@ID integer outputASSELECT TOP 1 @ID = [ID]FROM Queue WITH (READPAST, XLOCK)WHERE (Status = 0) AND (JobID = @JobID)RETURNGO
View 6 Replies View RelatedI know we are not allowed to benchmark SQL Server but..... It would be nice to have material to present which demonstrates the performance gains using a queue compared to insert/delete in a SQL table.
Logically it seems faster to use a queue due to the conversation grouping locking and the service broker itself. But there seems to be some overhead involved just to manage these queues that the service broker has to perform.
I am sure we are not unique with the choice to figure out if we will get a boost in performance using SQL a queue between services rather than a table to queue data. What is available to help understand the performance gains of using a queue?
I'm trying to flesh out a good queue table design with our dev team.So here is a general overview of the scenario. First an application will hit a WebAPI and grab any updates to Content and store those ID's in SQL (queue table). Next is the fun part, different multi threaded apps will process ID's from the queue. One app will make updates to the data in a different SQL DB while the other will update an index (likely Elastic).
Obviously, we don't want multiple threads working on the same items. One strategy could be to use UPDLOCK & READPAST query hints. However, I'm not sure about the reliability or performance of this solution. I just started looking into setting up a service broker but that would be completely unfamiliar territory for me. Also I can see how a broker might work well within the instance but how would that work with the application making updates to Elastic?
Hi all
We are writing a web-based multi-user call centre application application.
we are getting concurrency problems as you would expect with a multiuser application.
the application is made for callers who will bring up a different contact to call based on some predefined priority. now because the algorithm that prioritises the contacts takes a good 2 seconds to run, if 2 different caller request for the next prioritised contact, they will retrieve the same contact.
The only way that we think can resolve this problem is by building a queue. The queue would be implemented as a table, the particular implementation of this queue would be, when ever someone retrieves an entry from the queue, a background process will go on and generate a new queued item, i.e. in a FIFO manner. So that's how we think we should implement the queue.
Now come the question how to implement it. My idea is to have row level locking and a trigger to remove queue items from the queue. so that once one caller have looked at one of the item in the queue, another user can't look at the same item.
Any suggestions as to how i might be able to avoid concurrency problems?
What do you all think of my idea of implementing the FIFO queue?/
Is it possible to do row level locking in such a way that other users won't even be able to read the locked entry??
James :)
Trying to create a report... Report should show * documents on hold then depending on the "on-hold type" look in the corresponding table and SELECT a few fields. Here is what I have. Where do I SET the @profile variable to return the profile from my queue table?
DECLARe
@profilevarchar(256)
SELECT
q.[profile],q.on_hold,q.on_hold_message,q.dbc_state
FROM
QueueASq
[code]...
hey all you database guru's hopefully someone can lend some insight as to a little table design problem I have.
Basically I've got a system in place to authorize users to access a website typical username password stuff. The table contains a list of users and there passwords plus the auth level and a few other tid bits that aren't really important enough to into detail about them here. What I want to do is add a messaging system to this, I think I could probably figure out a way to do this half decent if I setup a seperate table for each user to add a row to the table for every message entry than in my asp.net code have it delete everything but the last 10 entries every time a user logs on. However I would much prefer a way that I didn't have to setup a whole new table for each user just for messaging purposes, maybe store something like a list in one of the database cell's kind of like .nets generic.list or better yet generic.queue, I would also like a way if it's possible without too much work to have the table automatically delete the oldest message every time a new message is received if there's already 10 messages existing for the user.
Anyways hopefully someone has some experience in setting up a system like this, I don't really require any code samples I can code it all myself (other than the database code to automatically remove entry's, I'm not a database guy) if someone could just explain a way to accomplish what I'm trying to do, or if someone has a different more convenient way of doing this I would be up for suggestions
Thanks in advance for any help offered, I do appreciate it
In a situation where messages are coming in faster than they can be processed, at what point will service broker start up another queue_reader? Also, how do you prevent table locking if part of the processing of that message involves inserting or updating data in a table? We are experiencing this problem because of the high number of messages coming through, and I'm not sure what the best solution is - does service broker have some built-in support for preventing contention on a table when multiple readers are running? Or maybe a pattern that can be used to get around it?
I have a requirement where once we create a new record in a table, we submit a query to fetch some data and save it in one of the columns of the newly created record. The main requirement is that the server where we fetch the data from can be down for sometime for regular maintenance and we do not want to loose the fetch query in that process. Is there a way we can implement this?
Thanks.
Hi Folks,
I was testing my error handling and purposefully failed some messages. Automatic posion message detection kicked in and disabled my queue. I tried the following, one at a time to enable it again but it doesn't work:
ALTER QUEUE MigrationQueue WITH STATUS = ON;
ALTER QUEUE MigrationQueue WITH STATUS = ON, ACTIVATION (STATUS = ON);
I would have thought the first line would've worked but I get the following when trying to receive...
The service queue "MigrationQueue" is currently disabled.
Help.
Hello!
I am running a basic SSB queue setup (more or less the Hello World example)and running into the following error message:
Transaction (Process ID 120) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
At first, I thought it was because I had the Initiator and Target Services on the same Queue, but I get this error even when I separate the two Services onto two Queues. This happens when I run more than one Target application receiving messages from the Target Queue.
Does anybody have any idea what could be happening here? Am I not allowed to set up more than one receiver?
Thanks --
Robert
I have a queue that, after running fine for several days will mysteriously turn off. It doesn't seem to be related to a poison message because I can restart the queue and processing resumes just fine. What are all the scenarios that would cause a queue to turn itself off, so I can 1) take preemptive action to prevent it from happening in the first place and 2) respond appropriately when it occurs.
Also, how to properly setup and verify that the BROKER_QUEUE_DISABLED is working properly. This is the SQL that I have so far, but is there a more direct way to raise the event other than writing an activated stored procedure that rolls back 5 times?
CREATE QUEUE [EventNotificationsQueue];
GO
CREATE SERVICE [EventNotificationsService]
ON QUEUE [EventNotificationsQueue]
([http://schemas.microsoft.com/SQL/Notifications/PostEventNotification]);
GO
CREATE EVENT NOTIFICATION [QueueDisabled]
ON QUEUE [MyQueue]
FOR BROKER_QUEUE_DISABLED
TO SERVICE 'EventNotificationsService', 'MyDatabase';
GO
HI There
My activated proc is rolling back the transaction and putting the message abck on the queue infinately ?
Normally it disabled the queue after a few rollbacks, i can see in the sql log that it just keeps rolling back and re-activating thousands of times.
It only stops when i disable activation on the queue.
WHy is the queue not disabling ?
Thanx
Hi There
I have sent messages and they are all sitting in the transmission queue with a blank status, why is service broker not trying to send them ? They are no errors in the sql log. BOL says this is blank when it has not tried to send the message ? Service broker is definately activated in the database.
How do i force sql server to send anything in the transmission que ?
I have no idea what is wrong or where to check ?
Thanx
Running transactional replication, dedicated server for distributor. While performance in terms of latency is excellent (usually 1 sec, almost never higher than 4) the disk queue length on the distributor is extremely high (over 6 usually). Is this typical? On any other server I would be very concerned, but cpu and memory usage are excellent and as said, latency is good. what is recommended config for distributor? others see high queue length?
View 2 Replies View RelatedHi,
I have created Queue with the following syntax. But it is not getting activated itself. What I have to do to get it activated itself, and what could be the frequency by default.
CREATE QUEUE NewCustomerQueue
WITH ACTIVATION
(PROCEDURE_NAME = prProcessNewCustomers,
STATUS = ON,
MAX_QUEUE_READERS = 1,
EXECUTE AS SELF)
GO
If I execute the prProcessNewCustomers procedure manually it is showing that the Queue has been activated. What change I have to make in the syntax to get it activated itself.
Actually I have two scenarios in my requirement,
1. One Queue processing immediately when it receives data (Order Processing)
2. Another Queue, Process when the server is idle i.e., off-peak time (for mailing)
What syntax I have to use for these.
Please help.
Thanks in advance
Babu
Howdy,
This is a follow on from a previous post
http://www.dbforums.com/t984271.html
And now I have found something interesting :
(1) When I was monitoring the SystemProcessor Queue locally ( Via a term server login onto the box ) I would see a queue of 3-4. If I monitor the same parameter from a remote PC, I see a Processor Queue of 1 - why?
The box had 1 GB RAM ( SQL used 500 MB and had 250 MB free according to Task manager ).
(2)
I have another almost identical box that has same CPU but twice ammount of RAM ( 2 GB ) but has SystemProcessor Queue of almost
0 - why?
All other parameters for Disk, IO etc are fine.
Cheers,
SG
avg disk queue length is 100%. Any ideas?
=============================
http://www.sqlserverstudy.com
Again one client with SAN EMC and again performace is several times worse then you can have with a cheap and primitive IDE drive... :(
Anyway, my question.
I am monitoring many parameters, including Avg Read Queue and Avg Write Queue.
So if I ReadQueue=3 and WriteQueue=7, what does it mean?
Scenario 1, there are 2 different Queues (R- read request, W - write request):
Windows --> Device
Read. Queue: R:R:R
Write Queue: W:W:W:W:W:W:W
Scenario 2:
Windows --> Device
Common Queue: W.R.W.W.R.W.W.R.W.W
In other words, if SQL server flushes writes (TRAN COMMIT or CHECKPOINT), generating hundreds or even thousands of write requests in few milliseconds, so Queue grows to 100-300 for a second or so, are read requests locked during that time?
Hi,
I created several queues on the default filegroup and I was wondering how can I change a queue filegroup?
Thanks,
Shai.
I am doing backup job using windows 2003 backup. I hv several scheduled backup job. And sometimes, when the first backup job not yet finish backing up, the second backup job will automatic starts at the set time resulting 2 backup jobs at 1 time. I need to know whether in SQL2000/2005, we can set the tsql command to do the job queue for the windows 2003 backup job, meaning if the 1st backup job is still ongoing, the 2nd backup job will hv to wait even though the set time for the job was already expired. When the 1st job done, then only the 2nd job started.
Appreciate the feedback..
tq
hi everybody,
i am working on a SSIS package that creates Multiple file from a table depending on a particular field.
now the requirement from the target system is i have send the files directly to the particular serverrs.
for that i have to post my files in msmq and then use some other technology to achieve this.
if anybody has some ideas on this it will be very helpful for me.
thanks in advance
srikanth
My normal scenario is tills sending live slaes to head office via service Broker queue. Sales are sent as soon as sale is made(normally tills on adsl lines), a loss of a Link to head office will still allow a sale to be made with the sale sitting in the transmission queue on the till but has not been commited at HO to adjust stock etc, until the link is back up. My worry is link goes down with several sales siting in the queue and then hardware failure. Is their a mechanism to backup the Transmission queue in the case of no link?
If you could back up the transmission queue at a till could i then take those messages and copy them into head offices queue for processing, this is in the case of my link to the till was down for a whole day but needed the daily sales from the shop,could i get the back up at end of the day and apply the messages at head office?
Any infomation on what would be the best way to recover from this event would be gratefully accepted.
any examples on a heartbeat to check the link status that is of very low cost between HO and tills bearing in mind HO needs to maintain status for large number of tills.
Hello,
I'm currently looking for a way to decently pauze the processing of messages by activation procedures. When I pauze the processing I want to be sure that the running activation procedures terminated.
I thought I could do this using ALTER QUEUE WITH ACTIVATION (STATUS = OFF) and sys.dm_broker_activated_tasks.
This is what I've tested: I set up one queue with an activation procedure attached to it that receives one message, then waits for one minute, then writes that message into a table. The queues max queue readers is set to 2. Now, I put 5 messages in the queue, 2 procs pop up in sys.dm_broker_activated_tasks. I run the alter queue statement shown above and check sys.dm_broker_activated_tasks again. It is empty so allegedly there are no tasks running. I check the table in which the messages are written by the activation procs and after some time I can see the results of both activation procedures which were according to sys.dm_broker_activated_tasks not running (but of course they were). The rest of the process works as expected, the procs reactivate upon reenabling the activation status and all is well.
Is there another way of checking the running activation procedures that checking sys.dm_broker_activated_tasks?
Is this intended behaviour or just a bug?
Hi There
I am having alot of trouble with the order in which messages are being delivered, here is my scenario.
I have a transaction action table :
message 1 - xml schema A.
message 2 - xml schema A.
message 3 - xml schema B.
message 4 - xml schema D.
I have the following SP:
BEGIN TRAN
BEGIN DIALOG CONVERSATION
SEND ON CONVERSATION message 1
BEGIN DIALOG CONVERSATION
SEND ON CONVERSATION message 2
COMMIT
AT the target i get message 1 in the queue first them message 2. But then i try this.
BEGIN TRAN
BEGIN DIALOG CONVERSATION
SEND ON CONVERSATION message 3
BEGIN DIALOG CONVERSATION
SEND ON CONVERSATION message 4
COMMIT
At the target i get message 4 first it has the lower queuing_order on the queue, somehow it got ont he queue first.
I have tried turning validation off on the mssage types as i thought it may have something to do with the xml, but same problem.
I then tried to do a commit after each message like this:
BEGIN TRAN
BEGIN TRAN
BEGIN DIALOG CONVERSATION
SEND ON CONVERSATION message 1
COMMIT
BEGIN TRAN
BEGIN DIALOG CONVERSATION
SEND ON CONVERSATION message 2
COMMIT
COMMIT
But message 4 still gets on the queue first . WTF is going on ?
The only way i can get this to work is like this.
BEGIN TRAN
BEGIN DIALOG CONVERSATION
SEND ON CONVERSATION message 1
WAITFOR DELAY '00:00:01'
BEGIN DIALOG CONVERSATION
SEND ON CONVERSATION message 2
COMMIT
Why does this work ? and why do i not have this issue with message 1 and 2? If it has soemthing to do with the xml what is it?
Please help.
I am running SQL server 2000 SP4 on a server with 2 Dual core 4G processors with data attached via a SAN>
I have a 70G database with 10 users that is giving attrocious performance. I have just tried to run a count(*) accross a couple of tables and am still waiting for the results 15 mins later. When I look at the disk queue it is around 50/60. I thought the target for this was around 2. I am sure that the hardware that we have in place is capable of running this db. However I`m not sure how to fully analyse what is going wrong here.
Any tips would be greatfully received.
Si
How to clean all messages of a queue ?
View 5 Replies View RelatedIs it possible to receive from a queue by a conversation handle? In the documentation there is an example that show you how to do it. Yet, if you "read" the whole document it says that the conversation handle can not be an expression.
The WHERE clause of the RECEIVE statement may only contain search conditions that use conversation_handle or conversation_group_id. The search condition may not contain any of the other columns in the queue. The conversation_handle or conversation_group_id may not be an expression.
Here is what I'm trying to do:
;RECEIVE TOP(1) @MsgBody = CAST(message_body as XML)
FROM ProcessingLetters
WHERE conversation_handle = @Conversation_Handle
It doesn't seem to matter if I use RECEIVE or SELECT. It will return nothing.
I've even tried this:
where cast(Conversation_Handle as varchar(100)) = cast(@Conversation_Handle as varchar(100))
Why am I doing this? I've put something into the queue to let me know that something is processing. When it is done I want to pull it out and end the conversation.
So is the WHERE conversation_handle = @Conversation_Handle supposed to work?
Thanks.
Hi There
I am guessing defining multiple services on the same queue is basically for providing different services to outside sources while using 1 queue, obviously as long as the contracts and activated sp logic applies to all services defined on that queue.
I am defining a queue per service, is this right or wrong or irrelevant ? It just works better for me in terms of manageability.
I just want to make sure there is no "best practice" reason for doing either or ? Is it just a matter of preference?
Any comments?
Thanx
My transmission queue has lots of messages that will never, ever be delivered because the transmission_status = "The session keys for this conversation could not be created or accessed. The database master key is required for this operation."
How can I purge the transmission queue to get rid of this junk?
Hi,
I'm new to the service broker service. All I want to do is to monitor the queue status. If the queue is disabled, send me an email alert.
Can you let me know what's the best way to accomplish it?
Thanks,
Jia
Hello!
In running some performance tests on a Queue using a message size of ~5KB, we found that we can process (SEND and RECEIVE) on the order of 600 - 800 messages / second. However, we have found that INSERTs of new messages to the Queue appear to take great precedence over DELETEs of received messages from the queue. In particular, we found that during heavy use the total size of the Queue (as determined using the sp_spaceused procedure) equals about the number of total messages processed, not the number of messages on the queue.
When we stop sending messages, the overall size of the Queue table appears to decrease slowly, so there is a background process that is obviously doing some work there to clean up the received messages from the Queue. What I would like to know is if we can affect that background process in any way so that the messages are cleared out more quickly. The performance has been determined to suffer appreciably once the Queue size grows to greater than about 3GB in size. We also notice timeouts on the RECEIVE statements when the Queue size is that large.
Thanks for any help --
Robert