forked from DIRACGrid/DIRAC
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathrelease.notes
5219 lines (3802 loc) · 178 KB
/
release.notes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
[v6r15-pre24]
Removed general "from DIRAC.Core.Utilities import *" in the top-level __init__.py
Made service handlers systematically working with unicode string arguments
Added requirements.txt and Makefile in the root of the project to support pip style installation
DIRAC documentation moved to the "docs" directory if the DIRAC project from the
DIRACDocs separate project.
*Accounting
CHANGE: INTEGER -> BIGINT for "id" in "in" accountingDB tables
*Core
NEW: The S_ERROR has an enhanced structure containing also the error code and the call
stack from where the structure was created
NEW: DErrno module to contain definitions of the DIRAC error numbers and standard
descriptions to be used from now on in any error code check
CHANGE: gMonitor instantiation removed from DIRAC.__init__.py to avoid problems in
documentation generation
CHANGE: removed Core.Utilities.List.sortList (sorted does the job)
CHANGE: removed unused module Core.Utilities.TimeSeries
NEW: dirac-install - makes us of the DIRAC tar files in CVMFS if available
NEW: dirac-install-client - a guiding script to install the DIRAC client from A to Z
CHANGE: dirac-install - when generating bashrc and cshrc scripts prepend DIRAC paths
to the ones existing in the environment already
NEW: MJFTimeLeft - using Machine JOb features in the TimeLeft utility
FIX: BaseClient - only give warning log message "URL banned" when one of the
service URLs is really banned
CHANGE: DISET components - improved logic of service URL retries to speedup queries
in case of problematic services
NEW: dirac-rss-policy-manager - allows to interactively modify and test only the
policy section of Dirac.cfg
FIX: XXXTimeLeft - do not mix CPU and WallTime values
FIX: ComponentInstaller - longer timeout for checking components PID (after restart)
CHANGE: Proxy - in executeWithUserProxy() when multiple DNs are present, try all of them
CHANGE: List utility - change uniqueElements() to be much faster
NEW: Platform - added getPlatform() and getPlatformTuple() utilities to evaluate lazily the
DIRAC platform only when it is needed, this accelerates DIRAC commands not needing
the platform information.
*Configuration
NEW: GOCDB2CSAgent agent to synchronize GOCDB and CS data about perfSONAR services
NEW: VOMS2CSAgent to synchronize VOMS user data with the DIRAC Registry
CHANGE: ConfigurationData - lazy config data compression in getCompressedData()
*Framework
CHANGE: SystemAdministratorIntegrator - make initial pinging of the hosts in parallel
to speed up the operation
CHANGE: InstalledComponentsDB - table to cache host status information populated
by a periodic task
NEW: ComponentInstaller Client class to encapsulate all the installation utilities
from InstallTools module
NEW: SystemAdministratorClientCLI - added uninstall host command
NEW: SystemAdministratorClientCLI - added show ports command
NEW: SystemAdministratorHandler - added getUsedPorts() interface
NEW: SystemAdministratorHandler - show host command shows also versions of the Extensions
NEW: InstalledComponentsDB - added Extension field to the HostLogging table
FIX: SystemLoggingDB - fixed double creation of db tables
*Accounting
FIX: DataStoreClient - Synchronizer based decorators have been replaced with a simple
lock as they were blocking addRegister() during every commit();
*RSS
NEW: CE Availability policy, closing #2373
CHANGE: Ported setStatus and setToken rpc calls to PublisherHandler from LHCb implementation
NEW: E-mails generated while RSS actions are now aggregated to avoid avalanches of mails
NEW: dirac-rss-sync is also synchronizing Sites now
*DMS
CHANGE: FileCatalogClient - make explicit methods for all service calls
CHANGE: DataManager, StorageElement - move physical accounting the StorageElement
CHANGE: FileCatalog - added recursive changePathXXX operations
CHANGE: FileCatalog contained objects have Master attribute defined in the CS. Extra check of eligibility of the catalogs specified explicitely. No-LFN write methods return just the Master result to be compatible with the current use in the clients.
CHANGE: Removed LcgFileCatalogXXX obsoleted classes
NEW: ConsistencyInspector class to perform data consistency checks between
different databases
CHANGE: FileCatalog(Client) - refactored to allow clients declare which interface
they implement
NEW: FileCatalog - conditional FileCatalog instantiation based on the configured
Operations criteria
*TS
CHANGE: TransformationDB table TaskInputs: InputVector column from BLOB to MEDIUMTEXT
FIX: TaskManager - fix bug in case there is no InputData for a task, the Request created
for the previous task was reassigned
NEW: TaskManager - possibility to submit one bulk job for a series of tasks
*WMS
NEW: TaskQueueDB - possibility to present requirements in a form of tags from the
site( pilot ) to the jobs to select ones with required properties
FIX: JobWrapper - the InputData optimizer parameters are now DEncoded
CHANGE: JobAgent - add Processors and WholeNode tags to the resources description
CHANGE: SiteDirector - flag to always download pilot output is set to False by default
FIX: SiteDirector - using PilotRunDirectory as WorkingDirectory, if available at the CE
level in the CS. Featire requested in issue #2746
NEW: MultiProcessorSiteDirector - new director to experiment with the multiprocessor/
wholeNode queues
CHANGE: JobMemory utility renamed to JobParameters
CHANGE: CheckWNCapabilities pilot command changed to get WN parameters from the
Machine Job Features (MJF) - NumberOfProcessors, MaxRAM
NEW: JobManager, ParametricJob - utilities and support for parametric jobs with multiple
parameter sequences
NEW: SiteDirector - added logic to send pilots to sites with no waiting pilots even if
the number of already sent pilots exceeds the number of waiting jobs. The functionality
is switched on/off by the AddPilotsToEmptySites option.
*RMS
FIX: Request - fix for the case when one of the request is malformed, the rest of
the requests could not be swiped
FIX: ReqProxyHandler - don't block the ReqProxy sweeping if one of the request is buggy
CHANGE: ReqProxyHandler - added monitoring counters
NEW: ReqProxyHandler - added interface methods to list and show requests in a ReqProxy
*Resources
FIX: SRM2Storage - do not add accounting to the output structure as it is done in
the container StorageElement class
CHANGE: Add standard metadata in the output of all the Storage plugins
*Interfaces
NEW: Job API - added setParameterSequence() to add an arbitrary number of parameter
sequences for parametric jobs, generate the corresponding JDL
*tests
NEW: The contents of the TestDIRAC package is moved into the tests directory here
[v6r14p30]
*Framework
BUGFIX: MonitoringHandler - in deleteActivities() use retVal['Message'] if result is not OK
*Resources
FIX: XROOTStorage - in getFile() evaluate file URL without URL parameters
in __putSingleFile() use result['Message'] in case of error
*RMS
FIX: dirac-rms-cancel-request - fixed crash because of gLogger object was not imported
*TS
FIX: TransformationCLI - in resetProcessedFile() added check that the Failed dictionary
is present in the result of a call
[v6r14p29]
*Core
FIX: Time - skip the effect of timeThis decorator if not running interractively
*DMS
FIX: DataManager - in getFile(), select preferentially local disk replicas, if none disk replicas,
if none tape replicas
FIX: DataManager - avoid changing argument of public method checkActiveReplicas()
FIX: FTSAgent - wait 3 times longer for monitoring FTS jobs if Staging
*Accounting
CHANGE: Jobs per pilot plot is presented as Quality plot rather than a histogram
*WMS
CHANGE: dirac-wms-cpu-normalization - reduce memory usage by using xrange() instead of range()
in the large test loop
[v6r14p28]
*TS
FIX: TaskManager - protection against am empty task dictionary in
prepareTransformationTasks()
FIX: Test_Client_TransformationSystem - fixes ti run in the Travis CI
environment
*WMS
FIX: JobMemory - use urllib instead of requests Python module as the latter
can be unavailable in pilots.
[v6r14p27]
*Core
FIX: PlainTransport,SocketInfoFactory - fix for the IPv6 "Address family not supported
by protocol" problems
*Interfaces
NEW: Dirac.py - in ping()/pingService() allow to ping a specific URL
*Resources
FIX: LcgFileCatalogClient - convert LFN into str in __fullLfn to allow LFNs
in a unicode encoding
*WMS
FIX: JobWrapper - set the job minor status to 'Failover Request Failed'
if the failover request fails sending
*TS
FIX: TransformationDB - in getTransformationTasks(),getTaskInputVector
forward error result to the callers
FIX: TaskManager - in case there is no InputData for a task, the Request created
for the previous task was reassigned. This fixes this bug.
*tests
FIX: several fixes to satisfy on-the-fly unit tests with teh Travis CI service
[v6r14p26]
NEW: Enabled on-the-fly tests using the Travis-CI service
*Core
FIX: Subprocess - fix two potential infinite loops which can result in indefinite
output buffer overflow
*WMS
FIX: JobScheduling executor - check properly if staging is allowed, it was always True before
[v6r14p25]
*Core
FIX: Subprocess - more detailed error log message in case ov output buffer
overflow
*DMS
FIX: DataManager - fix for getActiveReplicas(): first check Active replicas before
selecting disk SEs
*Resources
FIX: StorageElementCache - fixes to make this class thread safe
FIX: StorageFactory - fix in getConfigStorageProtocols() to properly get options
for inheriting SE definitions
[v6r14p24]
*Accounting
FIX: Plots, JobPlotter - fix sorting by plot labels in case the enddata != "now"
*DMS
FIX: dirac-dms-user-lfns - add error message when proxy is expired
[v6r14p23]
*Interfaces
FIX: Job.py - setCPUTime() method sets both CPUTime and MaxCPUTime JDL parameters
for backward compatibility. Otherwise this setting was ignored by scheduling
*TS
BUGFIX: TaskManager - bug fixed in submitTransformationTasks in getting the TransformationID
[v6r14p22]
CHANGE: Multiple commands - permissions bits changed from 644 to 755
*Framework
FIX: UserProfileDB - in case of desktop name belonging to two different users we have
to use both desktop name and user id to identify the desktop
*WMS
BUGFIX: JobWrapperTemplate - bug fixed in evaluation of the job arguments
*TMS
CHANGE: TaskManager - added TransformationID to the log messages
[v6r14p21]
*DMS
CHANGE: dirac-admin-allow(ban)-se - allow an SE group to be banned/allowed
*SMS
FIX: RequestPreparationAgent - fix crash in execute() in case no replica information
available
*WMS
FIX: TaskQueueDB, PilotAgentsDB - escape DN strings to avoid potential SQL injection
FIX: JobWrapperTemplate - pass JobArguments through a json file to fix the case
of having apostrophes in the values
*TMS
FIX: TransformationAgent - in processTransformation() fix reduction of number of files
[v6r14p20]
*WMS
FIX: SandboxMetadataDB - escape values in SandboxMetadataDB SQL queries to accommodate
DNs containing apostrophe
[v6r14p19]
*Core
NEW: CLI base class for all the DIRAC CLI consoles, common methods moved to the new class,
XXXCLI classes updated to inherit the base class
FIX: Network - fix crash when path is empty string, fixes partly #2413
*Configuration
FIX: Utilities.addToChangeSet() - fix the case when comma is in the BDII Site description
followed by a white space, the description string was constantly updated in the CS
*Interfaces
FIX: Dirac.py - in retrieveRepositorySandboxes/Data - "Retrieved" and "OutputData" key values
are strings '0' in the jobDict when a repository file is read, need to cast it to int
*DMS
FIX: RegisterReplica - if operation fails on a file that no longer exists and has no
replica at that SE, consider the operation as Done.
*Resources
FIX: ARCComputingElement - bug fix in getJobOutput in using the S_ERROR()
[v6r14p18]
*Core
FIX: VOMSService - attGetUserNickname() can only return string type values
FIX: dirac-deploy-scripts - install DIRAC scripts first so that they can be
overwritten by versions from extensions
*Framework
FIX: dirac-populate-component-db - bug fixed to avoid duplicate entries in the
database
*TS
FIX: TaskManager - do not use ReqProxy when submitting Request for Tasks, otherwise
no RequestID can be obtained
*Interfaces
CHANGE: Dirac.py - increase verbosity of a error log message in selectJobs
*Resources
FIX: XROOTStorage - fixed KeyError exception while checking file existence
FIX: ARCComputingElement - in getJobOutput test for existence of an already
downloaded pilot log
[v6r14p17]
*Core
FIX: Service.py - use the service name as defined in the corresponding section in the CS
and not the name defined in service Module option. This fixes the problem with the
StorageElement service not interpreting properly the PFN name and using a wrong local
data path.
*Resources
CHANGE: ARCComputingElement - if the VO is not discoverable from the environment, use ARC API
call in the getCEStatus, use ldapsearch otherwise
[v6r14p16]
*Resources
CHANGE: ARC Computing Element automatically renew proxies of jobs when needed
[v6r14p15]
*Core
FIX: VOMS.py - Fixed bug that generates proxies which are a mix between legacy and rfc proxies.
*DMS
CHANGE: Allow selecting disk replicas in getActiveReplicas() and getReplicas()
*WMS
CHANGE: Use the preferDisk option in the InputData optimizer, the TransformationAgent and in the Interface splitter
[v6r14p14]
*Core
FIX: VOMS.py - return RFC proxy if necessary after adding the VOMS extension
*Configuration
FIX: Validate maxCPUTime and Site description value
*Resources
FIX: XROOTStorage - changes to allow third party transfers between XROOT storages
CHANGE: HTCondorCEComputingElement - the Condor logging can now be obtained in the webinterface;
SIGTERM (instead of SIGKILL) is send to the application in case jobs are killed by the host site;
when pilots are put in held status we kill them in condor and mark them as aborted.
*WMS
FIX: pilotCommands - fixes for intrepreting tags in the pilot
[v6r14p13]
*WMS
FIX: pilot commands CheckCECapabilities and CheckWNCapabilities were not considering the case of missing proxy
[v6r14p12]
*Core
FIX: allow a renormalization of the estimated CPU power
FIX: dirac-install: Make hashlib optional again (for previous versions of python, since the pilot may end up on old machines)
*Framework
FIX: allow to install agents with non-standard names (different from the module name)
*DMS
CHANGE: Consider files to reschedule and submit when they are Failed in FTS
*WMS
CHANGE: Move getCEStatus function back to using the ARC API
[v6r14p11]
*Core
FIX: XXXTimeLeft - set limit to CPU lower than wall clock if unknown
FIX: Logger - fix exception printing in gLogger.exception()
CHANGE: InstallTools - added more info about the process in getStartupComponentStatus()
CHANGE: Time - better report from timeThis() decorator
*DMS
CHANGE: FTSAgent - wait some time between 2 monitorings of each job
*WMS
NEW: pilotCommands - added CheckCECapabilities, CheckWNCapabilities commands
NEW: Added dirac-wms-get-wn-parameters command
*TS
NEW: Added dirac-production-runjoblocal command
FIX: TransformationAgent(Plugin) - clean getNextSite() and normalizeShares()
FIX: TransformationPlugin - added setParameters() method
*RSS
FIX: dirac-rss-sync - move imports to after the Script.getPositionalArguments()
*Resources
NEW: Added dirac-resource-get-parameters command
[v6r14p10]
*Configuration
FIX: Resources - getQueue() is fixed to get properly Tag parameters
*Framework
FIX: SecurityFileLog - fix for zipping very large files
*Resources
NEW: added dirac-resource-get-parameters command
*WMS
NEW: JobMonitoringHandler - add getJobsParameters() method
NEW: pilotCommands - added CheckCECapabilities, CheckWNCapabilities
NEW: Added dirac-wms-get-wn-parameters command
NEW: Matcher - generate internal tags for MaxRAM and NumberOfProcessors parameters
CHANGE: SiteDirector does not pass Tags to the Pilot
FIX: Matcher(Handler) - do not send error log message if No match found,
fixed Matcher return value not correctly interpreted
[v6r14p9]
*Core
FIX: BaseClient - enhance retry connection logic to minimize the overall delay
FIX: MessageBroker - fix of calling private __remove() method from outside
of the class
*Framework
BUGFIX: dirac-(un)install-component - bug in importing InstallTools module
*WMS:
FIX: JobWrapper - fix in getting the OutputPath defined in the job
*Resources
FIX: ARCComputingElement - add queue to the XRSL string
[v6r14p8]
*Core
FIX: XXXTimeLeft - minor fixes plus added the corresponding Test case
FIX: ReturnValues - fixes in the doc strings to comply with the sphinx syntax
FIX: SocketInfoFactory - in __sockConnect() catch exception when creating a
socket
*Interfaces
FIX: Job.py - fixes in the doc strings to comply with the sphinx syntax
*RSS
NEW: Configurations.py - new possible configuration options for Downtime Policies
*WMS
CHANGE: StatesAccountingAgent - retry once and empty the local messages cache
in case of failure to avoid large backlog of messages
CHANGE: SiteDirector - do not send SharedArea and ClientPlatform as pilot
invocation arguments
CHANGE: Matcher - allow matching by hosts in multi-VO installations
[v6r14p7]
*Core
CHANGE: XXXTimeLeft utilities revisited - all return real seconds,
code refactoring - use consistently always the same CPU power
*WMS
FIX: JobAgent - code refactoring for the timeLeft logic part
*Resources
BUGFIX: ComputingElement - get rid of legacy getResourcesDict() call
[v6r14p6]
*Configuration
FIX: Bdii2CSAgent - refresh configuration from Master before updating
FIX: Bdii2CSAgent - distinguish the CE and the Cluster in the Glue 1.0 schema
*DMS
CHANGE: FTSAgent - make the amount of scheduled requests fetched by the
FTSAgent a parameter in the CS
CHANGE: RMS Operations - check whether the always banned policy is applied for SEs
to a given access type
*RMS
FIX: RequestClient(DB,Manager) - fix bulk requests, lock the lines when selecting
the requests to be assigned, update the LastUpdate time, and expose the
assigned flag to the client
*WMS
FIX: JobAgent - when the application finishes with errors but the agent continues
to take jobs, the timeLeft was not evaluated
FIX: JobAgent - the initial timeLeft value was always set to 0.0
[v6r14p5]
*Core
FIX: X509Certificate - protect from VOMS attributes that are not decodable
*Resources
FIX: GFAL2_StorageBase - fixed indentation and a debug log typo
*WMS
BUGFIX: Matcher - only the first job was associated with the given pilot
FIX: pilotTools - 0o22 is only a valid int for recent python interpreters,
replaced by 18
[v6r14p4]
*Core
FIX: DictCache - fix the exception in the destructor preventing the final
cache cleaning
*Framework
FIX: SystemAdministratorClientCLI - corrected info line inviting to update
the pilot version after the software update
*DMS
FIX: FTSAgent - Add recovery of FTS files that can be left in weird statuses
when the agent dies
CHANGE: DataManager - allow to not get URLs of the replicas
CHANGE: FTSJob - keep and reuse the FTS3 Context object
*Storage
CHANGE: StorageManagerClient - don't fail getting metadata for staging if at
least one staged replica found
*WMS
FIX: CPUNormalization - protect MJF from 0 logical cores
FIX: JobScheduling - fix printout that was saying "single site" and "multiple sites"
in two consecutive lines
NEW: pilotTools,Commands - added CEType argument, e.g. to specify Pool CE usage
FIX: WatchDog - added checks of function return status, added hmsCPU initialization to 0,
removed extra printout
*Resources
FIX: GFAL2 plugins - multiple bug fixes
[v6r14p3]
*Core
BUGFIX: small bug fixed in dirac-install-component, dirac-uninstall-component
BUGFIX: VOMS - remove the temporary file created when issuing getVOMSProxyInfo
FIX: FileHelper - support unicode file names
FIX: DictCache - purges all the entry of the DictCache when deleting the DictCache object
*Framework
BUGFIX: dirac-populate-component-db - avoid return statement out of scope
*Interfaces
BUGFIX: Dirac - in submitJob() faulty use of os.open
*WMS
FIX: JobWrapper - avoid evaluation of OutputData to ['']
FIX: Matcher - the Matcher object uses a VO dependent Operations helper
CHANGE: JobAgent - stop agent if time left is too small (default 1000 HS06.s)
FIX: CPUNormalization - use correct denominator to get power in MJF
*Resources
FIX: ARCComputingElement - changed implementation of ldap query for getCEStatus
[v6r14p2]
*Core
FIX: Use GSI version 0.6.3 by default
CHANGE: Time - print out the caller information in the timed decorator
CHANGE: dirac-install - set up ARC_PLUGIN_PATH environment variable
*Framework
FIX: dirac-proxy-info - use actimeleft VOMS attribute
*Accounting
CHANGE: Removed SRMSpaceTokenDeployment Accounting type
*RSS
CHANGE: ResourceStatus - re-try few times to update the RSS SE cache before giving up
FIX: XXXCommand, XXXAction - use self.lof instead of gLogger
CHANGE: Added support for all protocols for SEs managed by RSS
*RMS
FIX: Request - produce enhanced digest string
FIX: RequestDB - fix in getDigest() in case of errors while getting request
*Resources
CHANGE: Propagate hideExceptions flag to the ObjectLoader when creating StorageElements
FIX: ARCComputingElement - multiple fixes after experience in production
*WMS
FIX: Pilot commands - fixed an important bug, when using the
dirac-wms-cpu-normalization script
[v6r14p1]
The version is buggy when used in pilots
*Core
NEW: dirac-install-component command replacing dirac-install-agent/service/executor
commands
*Resources
NEW: FileStorage - plugin for "file" protocol
FIX: ARCComputingElement - evaluate as int the job exit code
*RSS
FIX: CSHelpers - several fixes and beautifications
[v6r14]
*Core
NEW: CSGlobals - includes Extensions class to consistently check the returned
list of extensions with proper names
NEW: ProxyManagerXXX, ProxyGeneration, X509XXX - support for RFC proxies
NEW: ProxyInfo - VOMS proxy information without using voms commands
NEW: LocalConfiguration - option to print out license information
FIX: SocketInfo.py - check the CRL lists while handshaking
Configuration
NEW: ConfigurationClient - added getSectionTree() method
*Framework
NEW: InstalledComponentsDB will now store information about the user who did the
installation/uninstallation of components.
*Resources
NEW: ARCComputingElement based on the ARC python API
*RSS
FIX: Improved logging all over the place
*DMS
NEW: New FileCatalog SecurityManager with access control based on policies,
VOMSPolicy as one of the policy implementations.
NEW: lfc_dfc_db_copy - script used by LHCb to migrate from the LFC to the DFC with
Foreign Keys and Stored Procedures by accessing the databases directly
NEW: FileManagerPs.py - added _getFileLFNs() to serve info for the Web Portal
CHANGE: Moving several tests to TestDIRAC
*Interfaces
CHANGE: use jobDescription.xml as a StringIO object to avoid multiple disk
write operations while massive job submission
*WMS
FIX: Watchdog - review for style and pylint
CHANGE: Review of the Matcher code, extracting Limiter and Matcher as standalone
utilities
*Transformation
NEW: New ported plugins from LHCb, added unit tests
[v6r13p21]
*TS
FIX: Registering TargetSE for Standard TransformationAgent plugin
[v6r13p20]
*DMS
FIX: DMSHelpers - allow for more than one Site defined to be local per SE
*Resources
FIX: XRootStorage - fix in getURLBase()
[v6r13p19]
FIX: changes incorporated from v6r12p53 patch
[v6r13p18]
*WMS
FIX: JobWrapper - ported back from v6r14p9 the fix for getting OutputPath
[v6r13p17]
FIX: changes incorporated from v6r12p52 patch
[v6r13p16]
FIX: changes incorporated from v6r12p51 patch
[v6r13p15]
Included patches from v6r12p50 release
[v6r13p14]
*DMS
FIX: ReplicateAndRegister - fix a problem when a file is set Problematic
in the FC but indeed doesn't exist at all
*Resources
CHANGE: StorageFactory - enhance the logic of BaseSE inheritance in the
SE definition in the CS
*WMS
CHANGE: CPUNormalization, dirac-wms-cpu-normalization - reading CPU power
from MJF for comparison with the DIRAC evaluation
FIX: SiteDirector - create pilot working directory in the batch system working
directory and not in "/tmp"
[v6r13p13]
*DMS
BUGFIX: FileCatalogClient - bug fixed in getDirectoryMetadata()
[v6r13p12]
*Resources
FIX: StorageElement - bug fixed in inValid()
CHANGE: StorageFactory - do not interpret VO parameter as mandatory
[v6r13p11]
*DMS
BUGFIX: RemoveReplica - fix in singleRemoval()
FIX: dirac-dms-user-lfns - increased timeout
[v6r13p10]
CHANGE: Use sublogger to better identify log source in multiple places
*Core
CHANGE: Review / beautify code in TimeLeft and LSFTimeLeft
FIX: LSFTimeLeft - is setting shell variables, not environment variables,
therefore added an "export" command to get the relevant variable
and extract then the correct normalization
*Accounting
FIX: DataOperationPlotter - add better names to the data operations
*DMS:
FIX: DataManager - add mandatory vo parameter in __SEActive()
CHANGE: dirac-dms-replicate-and-register-request - submit multiple requests
to avoid too many files in a single FTS request
FIX: FileCatalog - typo in getDirectoryMetadata()
FIX: FileCatalog - pass directory name to getDirectoryMetadata and not file name
FIX: DataManager - in __SEActive() break LFN list in smaller chunks when
getting replicas from a catalog
*WMS
FIX: WMSAdministratorHandler - fix in reporting pilot statistics
FIX: JobScheduling - fix in __getSitesRequired() when calling self.jobLog.info
CHANGE: pilotCommands - when exiting with error, print out current processes info
[v6r13p9]
*Framework
FIX: SystemLoggingDB - schema change for ClientIPs table to store IPv6 addresses
*DMS
BUGFIX: DMSRequestOperationsBase - bug fix in checkSEsRSS()
FIX: RemoveFile - in __call__(): bug fix; fix in the BannedSE treatment logic
*RMS
BUGFIX: Operation - in catalogList()
BUGFIX: ReqClient - in printOperation()
*Resources
FIX: GFAL2_StorageBase - added Lost, Cached, Unavailable in getSingleFileMetadata() output
BUGFIX: GFAL2_StorageBase - fixed URL construction in put(get)SingleFile() methods
*WMS
FIX: InputDataByProtocol - removed StorageElement object caching
[v6r13p8]
*Framework
FIX: MonitoringUtilities - minor bug fix
*DMS
FIX: DataManager - remove local file when doing two hops transfer
*WMS
FIX: SandboxStoreClient - get the VO info from the delegatedGroup argument to
use for the StorageElement instantiation
*TMS
CHANGE: Transformation(Client,DB,Manager) - multiple code clean-up without
changing the logic
[v6r13p7]
*Core
NEW: X509CRL - class to handle certificate revocation lists
*DMS
FIX: RequestOperations/RemoveFile.py - check target SEs to be online before
performing the removal operation.
FIX: SecurityManager, VOMSPolicy - make the vomspolicy compatible with the old client
by calling in case of need the old SecurityManager
*Resources
BUGFIX: Torque, GE - methods must return Message field in case of non-zero return status
FIX: SRM2Storage - when used internaly, listDirectory should return urls and not lfns
*WMS
FIX: ConfigureCPURequirements pilot command - add queue CPU length to the extra local
configuration
FIX: JobWrapper - load extra local configuration of any
*RMS
FIX: RequestDB - fix in getRequestSummaryWeb() to suit the Web Portal requirements
*Transformation
FIX: TransformationManagerHandler - fix in getTransformationSummaryWeb() to suit
the Web Portal requirements
[v6r13p6]
*Core
FIX: X509Chain - use SHA1 signature encryption in all tha cases
*Resources
FIX: ComputingElement - take CPUTime from its configuration defined in the
pilot parameters
*WMS
FIX: SiteDirector - correctly configure jobExecDir and httpProxy Queue parameters
[v6r13p5]
*Resources
BUGFIX: Torque - getCEStatus() must return integer job numbers
FIX: StorageBase - removed checking the VO name inside the LFN
*WMS
FIX: InputData, JobScheduling - StorageElement needs to know its VO
*DMS
FIX: ReplicateAndRegister - Add checksumType to RMS files when adding
checksum value
FIX: DataManager - remove unnecessary access to RSS and use SE.getStatus()
FIX: DMHelpers - take into account Alias and BaseSE in site-SE relation
*RMS
FIX: Request - bug fixed in optimize() in File reassignment from one
Operation to another
*Transformation
FIX: TransformationDB - set derived transformation to Automatic
[v6r13p4]
*Core
FIX: VOMSService - treat properly the case when the VOMS service returns no result
in attGetUserNickname()
*DMS
FIX: FTSAgent, ReplicateAndRegister - make sure we use source replicas with correct
checksum
*RMS
FIX: Request - minor fix in setting the Request properties, suppressing pylint
warnings
CHANGE: File, Reques, Operation, RequestDB - remove the use of sqlalchemy on
the client side
*Resources
FIX: StorageElement - import FileCatalog class rather than the corresponding module
FIX: SLURM - proper formatting commands using %j, %T placeholders
FIX: SSHComputingElement - return full job references from getJobStatus()
*RSS
FIX: DowntimeCommand - checking for downtimes including the time to start in hours
*Workflow
CHANGE: FailoverRequest - assign to properties rather than using setters
*Transformation
FIX: TransformationClient(DB,Utilities) - fixes to make derived transformations work
[v6r13p3]
*DMS
FIX: DataManager - in putAndRegister() specify explicitly registration protocol
to ensure the file URL available right after the transfer
*Resources
FIX: SRM2Storage - use the proper se.getStatus() interface ( not the one of the RSS )
[v6r13p2]
*Framework
FIX: SystemAdministratorHandler - install WebAppDIRAC extension only in case
of Web Portal installation
CHANGE: dirac-populate-component-db - check the setup of the hosts to register
into the DB only installations from the same setup; check the MySQL installation
before retrieving the database information
*DMS
FIX: FTSAgent - fix in parsing the server result
FIX: FTSFile - added Waiting status
FIX: FTSJob - updated regexps for the "missing source" reports from the server;
more logging message
*Resources
FIX: SRM2Storage - fix in treating the checksum type
FIX: StorageElement - removed getTransportURL from read methods
*RMS
FIX: Request - typo in the optimize() method
[v6r13p1]
*Framework
CHANGE: SystemAdminstratorIntegrator - can take a list of hosts to exclude from contacting
*DMS
FIX: DataManager - fix in __getFile() in resolving local SEs
FIX: dirac-dms-user-lfns - sort result, simplify logic
*RMS
FIX: Request - Use DMSHelper to resolve the Failovers SEs
FIX: Operation - treat the case where the SourceSE is None
*WMS
FIX: WMSAdministratorHandler - return per DN dictionary from getPilotStatistics
[v6r13]
CHANGE: Separating fixed and variable parts of error log messages for multiple systems
to allow SystemLogging to work
*Core
FIX: MySQL.py - treat in detailed way datetime functions in __escapeString()
FIX: DictCache.get() returns now None instead of False if no or expired value
NEW: InstallTools - allow to define environment variables to be added to the component
runit run script
NEW: Changes to make the DISET protocol IP V6 ready
CHANGE: BaseClient - retry service call on another instance in case of failure
CHANGE: InnerRPCClient - retry 3 times in case of exception in the transport layer
CHANGE: SocketInfo - retry 3 times in case of handshaking error
CHANGE: MySQL - possibility to specify charset in the table definition
FIX: dirac-install, dirac-distribution - removed obsoleted defaults
NEW: Proxy utility module with executeWithUserProxy decorator function
*Configuration
NEW: CSAPI,dirac-admin-add-shifter - function, and script, for adding or modifying a
shifter in the CS
*Framework
FIX: NotificationDB - escape fields for sorting in getNotifications()
NEW: Database, Service, Client, commands for tracking the installed DIRAC components
*Interfaces
CHANGE: Dirac - changed method names, keeping backward compatibility
CHANGE: multiple commands updated to use the new Dirac API method names
*DMS
NEW: Native use of the FTS3 services
CHANGE: Removed the use of current DataLogging service
CHANGE: DataManager - changes to manage URLs inside StorageElement objects only
FIX: DataManager - define SEGroup as accessible at a site
CHANGE: DirectoryListing - extracted from FileCatalogClientCLI as an independent utility
CHANGE: MetaQuery - extracted from FileCatalogClientCLI as an independent utility
CHANGE: FileCatalogClientCLI uses external DirectoryListing, MetaQuery utilities
CHANGE: FileCatalog - replace getDirectoryMetadata by getDirectoryUserMetadata
NEW: FileCatalog - added new getDirectoryMetadata() interface to get standard directory metadata
NEW: FileCatalog - possibility to find files by standard metadata
NEW: FileCatalog - possibility to use wildcards in the metadata values for queries
NEW: DMSHelpers class
NEW: dirac-dms-find-lfns command
*WMS
NEW: SiteDirector - support for the MaxRAM queue description parameter
CHANGE: JobScheduling executor uses the job owner proxy to evaluate which files to stage
FIX: DownloadInputData - localFile was not defined properly
FIX: DownloadInputData - could not find cached files (missing [lfn])
*RMS
CHANGE: Removed files from the previous generation RMS
CHANGE: RMS refactored based on SQLAlchemy
NEW: ReqClient - added options to putRequest(): useFailoverProxy and retryMainServer
CHANGE: DMSRequestOperationsBase - delay execution or cancel request based on SE statuses
from RSS/CS
FIX: Fixes to make use of RequestID as a unique identifier. RequestName can be used in
commands in case of its uniqueness
*Resources
NEW: Computing - BatchSystem classes introduced to be used both in Local and SSH Computing Elements
CHANGE: Storage - reworked Storage Element/Plugins to encapsulate physical URLs
NEW: GFAL2_StorageBase.py, GFAL2_SRM2Storage.py, GFAL2_XROOTStorage.py
*RSS:
NEW: dirac-admin-allow(ban)-se - added RemoveAccess status
CHANGE: TokenAgent - added more info to the mail