- 7491 Views
- 4 replies
- 3 kudos
Resolved! Cannot use prepared statements with date functions
We are using PHP and the Databricks SQL ODBC driver and cannot run a query that users DATE functions with prepared statements. Sample script/docker setup here:https://github.com/rlorenzo/databricks_php/blob/odbc_prepare_error/test_connection.phpFor e...
- 7491 Views
- 4 replies
- 3 kudos
- 3 kudos
@Bilal Aslam​ We tried CAST and CONVERT and still getting the same error.
- 3 kudos
- 1082 Views
- 0 replies
- 1 kudos
Is there a better way for this matching?
I have an array:var arg = condColumnsKeyswith the elementsarg: Array[String] = Array(LOT_PREFIX, PS_NAME_BOOK_TEMPLATE_NAME, PS_NAME_PAGE_NAME, PS_NAME_FIELD_NAME)Desired outcome is to get the string "LOT_PREFIX" and store it in var ccLotPrefixMy fir...
- 1082 Views
- 0 replies
- 1 kudos
- 1889 Views
- 1 replies
- 1 kudos
Databricks Office Hours Our next Office Hours session is scheduled for April 27 2022 - 8:00 am PT. Do you have questions about how to set up or use Da...
Databricks Office HoursOur next Office Hours session is scheduled for April 27 2022 - 8:00 am PT.Do you have questions about how to set up or use Databricks? Do you want to learn more about the best practices for deploying your use case or tips on da...
- 1889 Views
- 1 replies
- 1 kudos
- 1 kudos
Just registered. Thank you and happy weekend.
- 1 kudos
- 3844 Views
- 1 replies
- 6 kudos
Resolved! Is it possible to use Autoloader with a daily update file structure?
We get new files from a third-p@rty each day. The files could be the same or different. However, each day all csv files arrive in the same dated folder. Is it possible to use autoloader on this structure?We want each csv file to be a table that gets ...
- 3844 Views
- 1 replies
- 6 kudos
- 6 kudos
@Stephanie Rivera​ , You can use pathGlobfilter, but you will need a separate autoloader for which type of file.df_alert = spark.readStream.format("cloudFiles") \.option("cloudFiles.format", "binaryFile") \.option("pathGlobfilter", alert.csv") \.load...
- 6 kudos
- 3078 Views
- 1 replies
- 5 kudos
 Announcing: Delta Live Tables !Â
Databricks is excited to announce the general availability of Delta Live Tables to you, our community. Anxiously awaited, Delta Live Tables (DLT) is the first ETL framework that uses a simple, declarative approach to building reliable streaming or ...
- 3078 Views
- 1 replies
- 5 kudos
- 5 kudos
Informative Content thanks for sharing.
- 5 kudos
- 2542 Views
- 0 replies
- 0 kudos
Delete the file
While exporting data from Databricks to Azure blob storage how can I delete the committed, started and success file? ​
- 2542 Views
- 0 replies
- 0 kudos
- 4897 Views
- 1 replies
- 2 kudos
Resolved! How to pass variables into query string?
I have two variables StartTimeStmp and EndTimeStmp, i am going to assign the Start timestamp to it based on Last Successful Job Runtime and EndTimeStamp would be current time of system.SET StartTimeStmp = '2022-03-24 15:40:00.000';SET EndTimeStmp = '...
- 4897 Views
- 1 replies
- 2 kudos
- 2 kudos
@Srinivas Gannavaram​ , in python:spark.sql(f""" SELECT CI.CORPORATE_ITEM_INTEGRATION_ID , CI.CORPORATE_ITEM_CD WHERE CI.DW_CREATE_TS < '{my_timestamp_variable}' ; """)
- 2 kudos
- 16300 Views
- 2 replies
- 3 kudos
- 16300 Views
- 2 replies
- 3 kudos
- 3 kudos
@Direo Direo​ , Yeah, this is a location inside your dbfs. The whole control is on you. Databricks do not delete something you keep in this location.
- 3 kudos
- 2625 Views
- 1 replies
- 5 kudos
- 2625 Views
- 1 replies
- 5 kudos
- 5 kudos
@Direo Direo​ , Yes, you use Merge syntax for that https://docs.delta.io/latest/delta-update.html.And is more efficient than overwriting if you want to update only part of the data, but you need to think about the logic of what to update so overwriti...
- 5 kudos
- 2368 Views
- 1 replies
- 4 kudos
Resolved! What's the best architecture for Structured Streaming and why?
I am building an ETL pipeline which reads data from a Kafka topic ( data is serialized in Thrift format) and writes it to Delta Table in databricks. I want to have two layersBronze Layer -> which has raw Kafka dataSilver Layer -> which has deserializ...
- 2368 Views
- 1 replies
- 4 kudos
- 4 kudos
@John Constantine​ , "Bronze Layer -> which has raw Kafka data"If you use confluent.io, you can also utilize a direct sink to DataLake Storage - bronze layer."Silver Layer -> which has deserialized data"Then use Delta Live Tables to process it to del...
- 4 kudos
- 815 Views
- 0 replies
- 0 kudos
G.I.S., Inc. is a distributor and fabricator of thermal and acoustical insulation systems for industrial, commercial, power, process, original equipme...
G.I.S., Inc. is a distributor and fabricator of thermal and acoustical insulation systems for industrial, commercial, power, process, original equipment manufacturers, plumbing and HVAC industries. In today's fast paced market, consumers have a multi...
- 815 Views
- 0 replies
- 0 kudos
- 2558 Views
- 1 replies
- 1 kudos
Resolved! "policy_id" parameter in JOB API
I can't find information about that parameter in https://docs.databricks.com/dev-tools/api/latest/jobs.htmlWhere is it documented?
- 2558 Views
- 1 replies
- 1 kudos
- 1 kudos
I believe it is just "policy_id". As an incomplete example the specification via API would be something like: { "cluster_id": "1234-567890-abd35gh", "spark_context_id": 1234567890, "cluster_name": "my_cluster", "spark_version": "9.1.x-scala2....
- 1 kudos
- 4589 Views
- 3 replies
- 4 kudos
Resolved! Write output of DataFrame to a file with tild ( ~) separator in Databricks Mount or Storage Mount with VM.
I need to write output of Data Frame to a file with tilde ( ~) separator in Databricks Mount or Storage Mount with VM. Could you please help with some sample code if you have any?
- 4589 Views
- 3 replies
- 4 kudos
- 4 kudos
@Srinivas Gannavaram​ , Does it have to be CSV with fields separated by ~?If yes is enough to add .option("sep", "~")(df .write .option("sep", "~") .csv(mount_path))
- 4 kudos
- 3885 Views
- 1 replies
- 2 kudos
Resolved! list users having access to scope credentials
Hello!How do I list all the users or groups having access to the key-vault backed scope credentials?Let's say, I have a scope called MyScope for which all the secrets are stored in MyKeyVault.I would like to see what users have access there and ideal...
- 3885 Views
- 1 replies
- 2 kudos
- 2 kudos
@Bartosz Wachocki​ , As secrets use ACL for the scope, you need to make an API call (can be via CLI also) to list ACL for the given scope >> 2.0/secrets/acls/list more info here https://docs.databricks.com/dev-tools/api/latest/secrets.html#list-secre...
- 2 kudos
- 7188 Views
- 2 replies
- 2 kudos
Bronze silver gold layers
Is there a best practise guide on setting up the delta lake for these 3 layers. ​I'm looking for document or scripts to run that will assist me.
- 7188 Views
- 2 replies
- 2 kudos
- 2 kudos
hi @Lloyd Vickery​ ,I would highly recommend to use Databricks Delta Live Tables (DLT) docs here https://databricks.com/product/delta-live-tables
- 2 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 555 | |
| 349 | |
| 287 |
