cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

RohitKulkarni
by Contributor II
  • 7893 Views
  • 6 replies
  • 6 kudos

External Table issue format in databricks

I am new to databricksI am trying to create a external table in databricks with below format :CREATE EXTERNAL TABLE Salesforce.Account( Id string ,  IsDeleted bigint,  Name string ,  Type string ,  RecordTypeId string ,  ParentId string ,  ShippingSt...

  • 7893 Views
  • 6 replies
  • 6 kudos
Latest Reply
AmitA1
Contributor
  • 6 kudos

Databricks is awesome if you have SQL knowledge....I just came across one of my problem in my project and databricks helped me a lot....like a use of low watermark to hold the load success date .....​

  • 6 kudos
5 More Replies
Anonymous
by Not applicable
  • 2481 Views
  • 4 replies
  • 0 kudos

Resolved! Safari problems after the maintenance on 12/9/2022

I'm experience some problems on Safari 15.3 ( MacOS )I would like to know if I am alone in this and how to fix ( if I can ) this.This is the Databricks SQLData science and Engineering. ( is this case Workflows).

Screen Shot 2022-09-13 at 11.58.39 AM Screen Shot 2022-09-13 at 12.00.19 PM
  • 2481 Views
  • 4 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

The problem is fixed, anything works as usual.

  • 0 kudos
3 More Replies
dulu
by New Contributor III
  • 3817 Views
  • 2 replies
  • 6 kudos

Is there a function similar to split_part, json_extract_scalar?

I am using spark_sql version 3.2.1. Is there a function that can replacesplit_part,json_extract_scalarare not?

  • 3817 Views
  • 2 replies
  • 6 kudos
Latest Reply
Ankush
New Contributor II
  • 6 kudos

pyspark.sql.functions.get_json_object(col, path)[source]Extracts json object from a json string based on json path specified, and returns json string of the extracted json object. It will return null if the input json string is invalid.​

  • 6 kudos
1 More Replies
FerArribas
by Contributor
  • 3242 Views
  • 4 replies
  • 3 kudos

How to import a custom CA certificate into the Databricks SQL module?

We need to be able to import a custom certificate (https://learn.microsoft.com/en-us/azure/databricks/kb/python/import-custom-ca-cert) in the same way as in the "data engineering" module but in the Databricks SQL module

  • 3242 Views
  • 4 replies
  • 3 kudos
Latest Reply
VaibB
Contributor
  • 3 kudos

You can try downloading it to DBFS and may be accessing it from there if you use case really needs that.

  • 3 kudos
3 More Replies
dulu
by New Contributor III
  • 3909 Views
  • 3 replies
  • 15 kudos

Resolved! How to count the number of campaigns per day based on the start and end dates of the campaigns in SQL Spark Databrick

I need to count the number of campaigns per day based on the start and end dates of the campaignsInput Table: Out needed (result):How do I need to write the SQL command in databricks to get the above result? thanks all

image 1 image 2
  • 3909 Views
  • 3 replies
  • 15 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 15 kudos

Just create an array with sequence, explode it, and then group and count:WITH cte AS (SELECT `campaign name`, explode(sequence(`Start date`, `End date`, interval 1 day)) as `Date` FROM `campaigns`) SELECT Count(`campaign name`) as `count uni...

  • 15 kudos
2 More Replies
Taha_Hussain
by Databricks Employee
  • 7226 Views
  • 5 replies
  • 5 kudos

Connect a BI Tool: How do I access my lakehouse data from my BI tool?

You can find a rich ecosystem of tools that allow you to work with all your data in-place and deliver real-time business insights faster.This post will help you connect your existing tools like dbt, Fivetran, PowerBI, Tableau or SAP to ingest, transf...

Screen Shot 2022-08-09 at 11.54.56 PM
  • 7226 Views
  • 5 replies
  • 5 kudos
Latest Reply
Axserv
New Contributor II
  • 5 kudos

Hello Taha, here is a fairly recent video provided by Databricks on conncecting Power BI : Demo Video: Connect to Power BI Desktop from Databricks - YouTube

  • 5 kudos
4 More Replies
Saikrishna2
by New Contributor III
  • 6466 Views
  • 7 replies
  • 11 kudos

Data bricks SQL is allowing 10 queries only ?

•Power BI is a publisher that uses AD group authentication to publish result sets. Since the publisher's credentials are maintained, the same user can access the data bricks database.•Number of the users are retrieving the data from the power bi or i...

  • 6466 Views
  • 7 replies
  • 11 kudos
Latest Reply
VaibB
Contributor
  • 11 kudos

I believe 10 is a limit as of now. See if you can increase the concurrency limit from the source.

  • 11 kudos
6 More Replies
Thanapat_S
by Contributor
  • 4013 Views
  • 3 replies
  • 5 kudos

Resolved! How could I export an Alert object for deployment to another Azure Databricks resource?

IntroductionI would like to use Alert feature for monitor job status (from log table) in Databricks-SQL.So, I have write a query in a query notebook (or object) to return result from log table. Also, I have set the alert object for monitoring and tri...

image image image
  • 4013 Views
  • 3 replies
  • 5 kudos
Latest Reply
Harun
Honored Contributor
  • 5 kudos

I am not seeing any direct option to export or version control the alert object other than the migrate option.https://docs.databricks.com/sql/api/queries-dashboards.html - check this link, it might help you in other way.

  • 5 kudos
2 More Replies
KVNARK
by Honored Contributor II
  • 1840 Views
  • 3 replies
  • 11 kudos

Is there any limitation in querying the no. of SQL queries in Databricks SQL workspace.

Is there any limitation in querying the no. of SQL queries in Databricks SQL workspace. 

  • 1840 Views
  • 3 replies
  • 11 kudos
Latest Reply
Rajeev_Basu
Contributor III
  • 11 kudos

1000 has been documented to be by default, though I have never checked the correctness.

  • 11 kudos
2 More Replies
Carlton
by Contributor
  • 5739 Views
  • 5 replies
  • 14 kudos

I would like to know why CROSS JOIN fails recognize columns

Whenever I apply a CROSS JOIN to my Databricks SQL query I get a message letting me know that a column does not exists, but I'm not sure if the issue is with the CROSS JOIN.For example, the code should identify characters such as http, https, ://, / ...

image
  • 5739 Views
  • 5 replies
  • 14 kudos
Latest Reply
Shalabh007
Honored Contributor
  • 14 kudos

@CARLTON PATTERSON​ Since you have given an alias "tt" to your table "basecrmcbreport.organizations", to access corresponding columns you will have to access them in format tt.<column_name>in your code in line #4, try accessing the column 'homepage_u...

  • 14 kudos
4 More Replies
Anonymous
by Not applicable
  • 2720 Views
  • 3 replies
  • 28 kudos

Resolved! Refresh Dashboard also make all related queried refresh?

Hi all,I have a quick currious. I know both query and dashboard page in Databricks SQL have refresh button to can them refresh. But one question it, when I'm in Dashboard page and click the refesh button. Does this thing also force every related quer...

  • 2720 Views
  • 3 replies
  • 28 kudos
Latest Reply
Anonymous
Not applicable
  • 28 kudos

Thanks all your support. It's totally clear for me now!!!

  • 28 kudos
2 More Replies
AnubhavG
by Contributor
  • 5594 Views
  • 8 replies
  • 29 kudos

Resolved! Are Python External UDFs supported in Databricks SQL warehouse?

I tried running a python UDF in the Databricks SQL warehouse but it did not run and gave the "Python UDF is not supported" error.Can i get a clear picture if the Python External UDFs are supported or not?

  • 5594 Views
  • 8 replies
  • 29 kudos
Latest Reply
youssefmrini
Databricks Employee
  • 29 kudos

It's a private preview and It will be supported only in PRO SQL Warehouse and the Serverless SQL Warehouse

  • 29 kudos
7 More Replies
sreedata
by New Contributor III
  • 5139 Views
  • 4 replies
  • 10 kudos

Resolved! Date field getting changed when reading from excel file to dataframe

The date field is getting changed while reading data from source .xls file to the dataframe. In the source xl file all columns are strings but i am not sure why date column alone behaves differentlyIn Source file date is 1/24/2022.In dataframe it is ...

  • 5139 Views
  • 4 replies
  • 10 kudos
Latest Reply
Pradeep_Namani
New Contributor III
  • 10 kudos

Hi Team, @Merca Ovnerud​ I am also facing same issue , below is the code snippet which I am using df=spark.read.format("com.crealytics.spark.excel").option("header","true").load("/mnt/dataplatform/Tenant_PK/Results.xlsx")I have a couple of date colum...

  • 10 kudos
3 More Replies
siva_thiru
by Contributor
  • 1167 Views
  • 0 replies
  • 6 kudos

Happy to share that #WAVICLE​  was able to do a hands-on workshop on #[Databricks notebook]​ #[Databricks SQL]​ #[Databricks cluster]​ Fundamentals wi...

Happy to share that #WAVICLE​  was able to do a hands-on workshop on #[Databricks notebook]​ #[Databricks SQL]​ #[Databricks cluster]​ Fundamentals with KCT College, Coimbatore, India.

Workshop Standee
  • 1167 Views
  • 0 replies
  • 6 kudos
Labels