- 2418 Views
- 2 replies
- 0 kudos
Resolved! Using FOR XML RAW in Spark SQL
How can I convert below SQL server query to spark SQL Query - SELECT distinct HashBytes('md5', (SELECT a, b, c for xml raw)) as xyzzy FROM table nameNeed help here community
- 2418 Views
- 2 replies
- 0 kudos
- 0 kudos
The Idea here was to create a unique id based upon the number of columns, I have used dense Rank to resolve this issueThanks
- 0 kudos
- 6053 Views
- 3 replies
- 0 kudos
What is the difference between SQS in S3 and AutoLoader in databricks?
I'm curious about the difference between using S3's SQS to set the queue url in spark's readStream option and AutoLoader reading from Cloudfiles.I would also like some advice on which one is better to use in which situation.(from a cost and performan...
- 6053 Views
- 3 replies
- 0 kudos
- 0 kudos
Using Spark readStream with SQS will be very similar to using CloudFiles with FileNotification mode (which also uses SQS on the backend).CloudFiles comes with some additional options compared to native Spark readStream:Common Auto Loader OptionsOne b...
- 0 kudos
- 5994 Views
- 6 replies
- 1 kudos
Command restrictions
Is there any possibility to restrict usage of specified commands (like mount/unmount or SQL grant) based on group assignment? I do not want everybody to be able to execute these commands.
- 5994 Views
- 6 replies
- 1 kudos
- 1 kudos
If you are ok for users to have only SQL syntax available ( no mounts ) , you can provision SQL warehouse for users , not clusters
- 1 kudos
- 10567 Views
- 2 replies
- 1 kudos
Resolved! com.microsoft.sqlserver.jdbc.SQLServerException: The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption.
Hi. I am trying to read from our Microsoft SQL Server from Azure Databricks via spark.read.jdbc() as described here: Query databases using JDBC - Azure Databricks | Microsoft Learn. The SQL Server is on an Azure VM in a virtual network peered with th...
- 10567 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Kazer ,Even if I use a new table name, I get the same error. Do you have any suggestions?Thanks,
- 1 kudos
- 2162 Views
- 0 replies
- 0 kudos
" Token is expiring within 30 seconds." when running a job using Databricks SDK
During attempt (in Azure DevOps environment) of running a job (and getting the result of the run) using Databricks SDK for Python, I've got this error:databricks.sdk.core.DatabricksError: Token is expiring within 30 seconds. (token expired). The stac...
- 2162 Views
- 0 replies
- 0 kudos
- 4178 Views
- 3 replies
- 0 kudos
Resolved! Community Edition Sign Up Issues
Hi, I have had to sign up for the Community Edition using my work email address so I can use this for training courses. I used my work mail address as it required me to enter my company name and position. The signup is successful but I have 2 quest...
- 4178 Views
- 3 replies
- 0 kudos
- 0 kudos
Hello DeveCommunity edition will be valid only for 14 days and you cannot login with the same credential until you convert the same in pay as you go model. Even though if you would like to use the Community edition with your personnel email id that a...
- 0 kudos
- 3697 Views
- 1 replies
- 0 kudos
Migrating jsonb data from Postgresql database to Databricks
Hi Team,I am trying to create a pipeline to incrementally ingest data from an RDS postgresql database which contains tables that have some columns of jsonb data type. I am currently using AWS DMS with CDC to first load the data into an S3 bucket as c...
- 3697 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello Karene, You can do the transformation in following manner from string to struct and refer to the example below: data =[('001','{"name":"bhupendra","zipcode":"260100"}')]schema = ['id','propertytype']df = spark.createDataFrame(data,schema)df.sho...
- 0 kudos
- 7802 Views
- 4 replies
- 1 kudos
urgent: apply changes into delta live tables
Hi @Retired_mod , @Avnish_Jain How can I implement cdc in sql dlt pipelines with a live table (not streaming) . I am trying to implement below where i am reading from external tables, loading data into bronze layer and then want to apply these change...
- 7802 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi Faisal, APPLY CHANGES INTO does not support a materialized view as a source, this must be a streaming table. Ideally, your bronze tables are append-only with the source providing data incrementally. If you do get revisions on previous records in y...
- 1 kudos
- 22036 Views
- 3 replies
- 4 kudos
Resolved! AttributeError: 'DataFrame' object has no attribute 'rename'
Hello, I am doing the Data Science and Machine Learning course. The Boston housing has unintuitive column names. I want to rename them, e.g. so 'zn' becomes 'Zoning'. When I run this command: df_bostonLegible = df_boston.rename({'zn':'Zoning'}, axi...
- 22036 Views
- 3 replies
- 4 kudos
- 4 kudos
If df_boston is a DataFrame, but you still face issues, try an alternative syntax: df_boston = df_boston.rename(columns={'zn': 'Zoning'}).Make sure df_boston is a proper DataFrame and you're using a recent version of Pandas.
- 4 kudos
- 3699 Views
- 1 replies
- 0 kudos
import * from ../my/relative/path
I have the following repository structure:/Repos/main/MyRepo/ -> run (folder) -> setup (folder) -> src (folder) -> main (notebook)Now, starting from a notebook in the "src" folder I need to run/import all variables defined in all notebooks stored in ...
- 3699 Views
- 1 replies
- 0 kudos
- 19757 Views
- 8 replies
- 0 kudos
error after updating delta table com.databricks.sql.transaction.tahoe.DeltaUnsupportedOperationException: Detected a data update
Hi,I have pipeline running. I have updated one file in delta table which is already processed. Now I am getting errorcom.databricks.sql.transaction.tahoe.DeltaUnsupportedOperationException: Detected a data update. This is currently not supported. If ...
- 19757 Views
- 8 replies
- 0 kudos
- 2013 Views
- 0 replies
- 0 kudos
- 2013 Views
- 0 replies
- 0 kudos
- 3477 Views
- 2 replies
- 0 kudos
Materialize View creation not supported from DLT Pipeline
Hi Team, I have a very basic scenario where I am using my custom catalog and want materialize view to get created from DLT table at the end of pipeline. The SQL used as below for the same.where "loom_data_transform" is a Streaming table. But pipeline...
- 3477 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey @abhijitnag are you sure your loom_data_transform was created as a STREAMING table? docs:https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-streaming-table.html
- 0 kudos
- 2196 Views
- 0 replies
- 0 kudos
DLT || Python || Aggregate Functions recomputing all the records
Hi all, I am building a realtime dashboard using Databricks Delta Live Tables Pipeline and using the following steps : - Bronze Table : Using the autoloader functionality provided by databricks, its incrementally ingesting new files records into a br...
- 2196 Views
- 0 replies
- 0 kudos
- 1266 Views
- 0 replies
- 0 kudos
Registered for Wrong certification
Hi, I am registered for databricks certification today at 23:00hr and I accidentally choose databricks data analyst certification instead of databricks data engineer associate.I have left several mails and submitted several forms to be able to fix th...
- 1266 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
