- 2999 Views
- 1 replies
- 0 kudos
Connecting to Serverless Redshift from a Databricks Notebook
Hello Experts, A new databricks user here. I am trying to access an Redshift serverless table using a databricks notebook. Here is what happens when I try the below code, df = spark.read.format("redshift")\.option("dbtable", "public.customer")\.opti...
- 2999 Views
- 1 replies
- 0 kudos
- 0 kudos
@arunak - we need to specify forward_spark_s3_credentials to true during read. This will help spark detect the credentials used to authenticate to the S3 bucket and use these credentials to r read from redshift.
- 0 kudos
- 5156 Views
- 1 replies
- 0 kudos
Write to csv file in S3 bucket
I have a pandas dataframe in my Pyspark notebook. I want to save this dataframe to my S3 bucket. I'm using the following command to save itimport boto3import s3fsdf_summary.to_csv(f"s3://dataconversion/data/exclude",index=False)but I keep getting thi...
- 5156 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @mh_db - you can import botocore library (or) if it is not found can do a pip install botocore to resolve this. Alternatively, you can maintain the data in a spark dataframe without converting to a pandas dataframe and while writing to a csv. you ...
- 0 kudos
- 3010 Views
- 0 replies
- 0 kudos
Custom Spark Extension in SQL Warehouse
I understand only a limited spark configurations are supported in SQL Warehouse but is it possible to add spark extensions to SQL Warehouse clusters?Use Case: We've a few restricted table properties. We prevent that with spark extensions installed in...
- 3010 Views
- 0 replies
- 0 kudos
- 9542 Views
- 8 replies
- 2 kudos
Activate spark extensions on SQL Endpoints
It would be possible to activate a custom extensions like Sedona (https://sedona.apache.org/download/databricks/ ) in SQL Endopoints?Example error:java.lang.ClassNotFoundException: org.apache.spark.sql.sedona_sql.UDT.GeometryUDT at org.apache.spark....
- 9542 Views
- 8 replies
- 2 kudos
- 2 kudos
@Retired_mod What is the right way to add custom spark extension to sql warehouse clusters?
- 2 kudos
- 15662 Views
- 1 replies
- 0 kudos
Resolved! Lakehouse Federation for SQL Server and Security Policy
We've been able to setup a Foreign Catalog using the following documentation:https://learn.microsoft.com/en-us/azure/databricks/query-federation/sql-serverHowever the tables that have RLS using a Security Policy appear empty. I imagine that this solu...
- 15662 Views
- 1 replies
- 0 kudos
- 0 kudos
Was a bit quick here, found out that the SUSER_NAME() of the query is of course the connection that was setup.So the User/Password defined here:Once I added that same user to the RLS logic I get the correct result. 
- 0 kudos
- 1595 Views
- 0 replies
- 0 kudos
DevOps Asset Bundle Deployment to Change the Catalog a Job Writes to
I am trying to set up CI/CD with azure devops and 3 workspaces, dev, test, prod using asset bundlesAll 3 workspaces will have their own catalog in unity catalog. I can't find a way to change which catalog should be used by the jobs and dlt pipelines ...
- 1595 Views
- 0 replies
- 0 kudos
- 2670 Views
- 1 replies
- 0 kudos
Support for Delta tables multicluster writes in Databricks cluster
Hello, We're using Databricks on AWS and we've recently started using Delta tables. We're using R.While the code below[1] works in a notebook, when running it from RStudio on a Databricks cluster we get the following error: java.lang.IllegalStateExce...
- 2670 Views
- 1 replies
- 0 kudos
- 0 kudos
Sorry, for being very late here - If you can not use multi write to false, we can try to split this table into separate tables for each stream.
- 0 kudos
- 6313 Views
- 1 replies
- 0 kudos
Cast string to decimal
Hello, can anyone help me with the below error.I'm trying to cast the string column into decimal. When I try to do that I'm getting the "Py4JJavaError: An error occurred while calling t.addCustomDisplayData. : java.sql.SQLException: Status of query a...
- 6313 Views
- 1 replies
- 0 kudos
- 2073 Views
- 0 replies
- 0 kudos
what config do we use to set row groups fro delta tables on data bricks.
I have tried multiples way to set row group for delta tables on data bricks notebook its not working where as I am able to set it properly using spark.I tried 1. val blockSize = 1024 * 1024 * 60spark.sparkContext.hadoopConfiguration.setInt( "dfs.bloc...
- 2073 Views
- 0 replies
- 0 kudos
- 60476 Views
- 6 replies
- 5 kudos
Resolved! Convert Date to YYYYMMDD in databricks sql
Hi,I have a date column in a delta table called ADate. I need this in the format YYYYMMDD.In TSQL this is easy. However, I can't seem to be able to do this without splitting the YEAR, MONTH and Day and concatenating them together.Any ideas?
- 60476 Views
- 6 replies
- 5 kudos
- 5 kudos
I'll share I'm having a variant of the same issue. I have a varchar field in the form YYYYMMDD which I'm trying to join to another varchar field from another table in the form of MM/DD/YYYY. Does anyone know of a way to do this in SPARK SQL without s...
- 5 kudos
- 14885 Views
- 5 replies
- 0 kudos
Resolved! Is it possible to control the ordering of the array values created by array_agg()?
Hi! I would be glad to ask you some questions.I have the following data. I would like to get this kind of result. I want `move` to correspond to the order of `hist`.Therefore, i considered the following query.```with tmp as (select * from (values(1, ...
- 14885 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @ThomazRossito This is a great idea. It can solve my problem.Thank you.
- 0 kudos
- 2827 Views
- 1 replies
- 0 kudos
Resolved! DLT refresh unexpectedly failing
We're hitting an error with a delta live table refresh since yesterday; nothing has changed in our system yet there appears to be a configuration error: { ... "timestamp": "2024-04-08T23:00:10.630Z", "message": "Update b60485 is FAILED.",...
- 2827 Views
- 1 replies
- 0 kudos
- 0 kudos
Apparently the `custom_tags` of `ResourceClass` is now extraneous -- removing it from config corrected our problem.
- 0 kudos
- 12136 Views
- 4 replies
- 3 kudos
Optimizing Delta Live Table Ingestion Performance for Large JSON Datasets
I'm currently facing challenges with optimizing the performance of a Delta Live Table pipeline in Azure Databricks. The task involves ingesting over 10 TB of raw JSON log files from an Azure Data Lake Storage account into a bronze Delta Live Table la...
- 12136 Views
- 4 replies
- 3 kudos
- 3 kudos
Hey @brian_zavareh , see this document. I hope this can help.https://learn.microsoft.com/en-us/azure/databricks/compute/cluster-config-best-practicesJust keep in mind that there's some extra cost from Azure VM side, check your Azure Cost Analysis for...
- 3 kudos
- 8130 Views
- 2 replies
- 1 kudos
Alerts and Dashboard
Hi Team,In my Databricks workspace, i have created an alerts using the query in such a way the schedule will run on daily basis and the results will get populated to dashboard. The results from dashboard will be notified via email, but i am seeing re...
- 8130 Views
- 2 replies
- 1 kudos
- 1 kudos
HI @Shas_DataE, Good Day! Could you please check and confirm if there are any special characters in the table column? At this moment, special characters are compatible with Excel. If yes then please drop the column that has that special character a...
- 1 kudos
- 4249 Views
- 2 replies
- 1 kudos
Resolved! date_format 'LLLL' returns '1'
Hi all,In my notebook, when I run my cell with following code%sqlselect date_format(date '1970-01-01', "LLL");I get '1', while I expect 'Jan' according to the dochttps://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html I would also expect t...
- 4249 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Retired_mod ,Turns out it was actually a Java 8 bug:IllegalArgumentException: Java 8 has a bug to support stand-alone form (3 or more 'L' or 'q' in the pattern string). Please use 'M' or 'Q' instead, or upgrade your Java version. For more details...
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
