cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

amitca71
by Contributor II
  • 4978 Views
  • 6 replies
  • 0 kudos

Customer-managed VPC with empty default security group

Hi,I use self managed VPC. when i use security group the jobs clusters are being created.if i make the default security group empty and create customized security group (required by soc2 to have empty default), with same definition of ingress (allow...

  • 4978 Views
  • 6 replies
  • 0 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 0 kudos

Hi @Amit Cahanovich​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.

  • 0 kudos
5 More Replies
eacuna
by New Contributor
  • 3866 Views
  • 4 replies
  • 1 kudos

how to disable delete option in data UI?

Hi, I need to remove the delete option from the data menu, how could I do it?

delete data option UI
  • 3866 Views
  • 4 replies
  • 1 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 1 kudos

Hi @Andres Acuña​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.

  • 1 kudos
3 More Replies
Mohan_Potla
by New Contributor
  • 2779 Views
  • 2 replies
  • 0 kudos

Link to "Lakehouse Fundamentals Accreditation (V2)" in Databricks Academy

Please share link to take Lakehouse Fundamentals Accreditation (V2) in Databricks Academy.

  • 2779 Views
  • 2 replies
  • 0 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 0 kudos

Adding @Vidula Khanna​ for visibility.

  • 0 kudos
1 More Replies
riccostamendes
by New Contributor II
  • 3452 Views
  • 3 replies
  • 0 kudos

Reading a adls gen 2 csv file from db using standard pandas

i have mounted the adls container where the data are, but I cannot read the files with pandas ('pd.read_csv') even though I have attached the prefix '/dbfs/' to the path . Instead if I use 'spark.read.csv', I have no problems.Anyone knows why this i...

  • 3452 Views
  • 3 replies
  • 0 kudos
Latest Reply
Debayan
Databricks Employee
  • 0 kudos

Hi, When you are not able to read the files, what is the error you get?

  • 0 kudos
2 More Replies
Hubert-Dudek
by Databricks MVP
  • 2385 Views
  • 2 replies
  • 6 kudos

One great advantage of using the dDatabricks VSCode extension is that it doesn't require a personal access token. All you need to do is sign in to...

One great advantage of using the dDatabricks VSCode extension is that it doesn't require a personal access token. All you need to do is sign in to your Azure account, and you're good to go!

Untitled
  • 2385 Views
  • 2 replies
  • 6 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 6 kudos

Indeed, thank you for sharing this!!!

  • 6 kudos
1 More Replies
sm1
by New Contributor III
  • 5327 Views
  • 5 replies
  • 2 kudos
  • 5327 Views
  • 5 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Suky Muliadikara​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.T...

  • 2 kudos
4 More Replies
jneira
by New Contributor III
  • 5681 Views
  • 1 replies
  • 2 kudos

Setting database/catalog section of databricks jdbc driver url does not make it default

Hi, first of all thanks for the jdbc driver software.I have been used it with almost no problems but today i tried to set a a a database by default other than hive_metastore. I have followed the instructions and the example from the pdf jdbc driver g...

  • 5681 Views
  • 1 replies
  • 2 kudos
Latest Reply
sanjevraj
New Contributor III
  • 2 kudos

For anyone looking for an answer I had to add ConnCatalog=<catalog-name> in the URL.It is in the documentation https://docs.databricks.com/integrations/jdbc-odbc-bi.html#databricks-jdbc under JDBC driver capabilities.

  • 2 kudos
shan_chandra
by Databricks Employee
  • 4036 Views
  • 1 replies
  • 1 kudos

Resolved! Adding spark_conf tag on Jobs API

using Jobs API, when we create a new job to run on an interactive cluster, can we add spark_conf tag and specify spark config tuning parameters?

  • 4036 Views
  • 1 replies
  • 1 kudos
Latest Reply
shan_chandra
Databricks Employee
  • 1 kudos

 spark_conf needs to be set prior to the start of the cluster or have to restart the existing cluster. Hence, the spark_conf tag is available only on the job_cluster. you may have to set the configs manually on the interactive cluster prior to using ...

  • 1 kudos
Hubert-Dudek
by Databricks MVP
  • 3060 Views
  • 3 replies
  • 5 kudos

Databricks added variable explorer to notebooks. It is beneficial when working on multiple cells and doesn&#39;t run them all. For example, we can see...

Databricks added variable explorer to notebooks. It is beneficial when working on multiple cells and doesn't run them all. For example, we can see variables that were initiated but from deleted cells (as that variables are still in cluster memory).

variables
  • 3060 Views
  • 3 replies
  • 5 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 5 kudos

Great post, thank you for sharing

  • 5 kudos
2 More Replies
Bie1234
by New Contributor III
  • 6628 Views
  • 3 replies
  • 4 kudos

Resolved! How to delete records that column have same value in another table?

delete from DWH.SALES_FACT where SALES_DATE in (select SALES_DATE from STG.SALES_FACT_SRC) AND STORE_ID in (select STORE_ID from STG.SALES_FACT_SRC)output : Error in SQL statement: DeltaAnalysisException: Nested subquery is not supported in the...

  • 6628 Views
  • 3 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @pansiri panaudom​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.T...

  • 4 kudos
2 More Replies
dhanu
by New Contributor
  • 2503 Views
  • 2 replies
  • 0 kudos

Fatal error: Python kernel is unresponsive

i have submitted around 90 job at a time to databricks, the job was running continuously for 2 hours after that i am getting fatal error Pyhon kernel is unresponsive.I am using Databricks runtime version : 11.2Cluster Configuration Details are given...

  • 2503 Views
  • 2 replies
  • 0 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 0 kudos

Hi @Dhanaraj Jogihalli​,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.

  • 0 kudos
1 More Replies
rajalakshmi9394
by New Contributor II
  • 5119 Views
  • 3 replies
  • 4 kudos

Resolved! QUERY_RESULT_ROWS without first row as trigger in SQL Alerts of databricks

Hi Team - In azure databricks sql alerts, I was able to use QUERY_RESULT_ROWS only if I'm selecting the trigger as first row. Is there a possibility to get count of number of rows and also the query result (both rows and columns to display the data a...

  • 5119 Views
  • 3 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Rajalakshmi Amara​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell u...

  • 4 kudos
2 More Replies
Ajay-Pandey
by Databricks MVP
  • 2778 Views
  • 1 replies
  • 6 kudos

Cluster policies now support limiting the max number of clusters per user can create Policy permissions allow you to set a max number of clusters per ...

Cluster policies now support limiting the max number of clusters per user can createPolicy permissions allow you to set a max number of clusters per user. This determines how many clusters a user can create using that policy. If the user exceeds the ...

image
  • 2778 Views
  • 1 replies
  • 6 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 6 kudos

Thank you for sharing

  • 6 kudos
Ajay-Pandey
by Databricks MVP
  • 4626 Views
  • 1 replies
  • 6 kudos

Variable explorer in Databricks With Databricks Runtime 12.1 and above, you can directly observe current Python variables in the notebook UI.To open t...

Variable explorer in DatabricksWith Databricks Runtime 12.1 and above, you can directly observe current Python variables in the notebook UI.To open the variable explorer, click in the right sidebar. The variable explorer opens, showing the value and ...

image Untitled
  • 4626 Views
  • 1 replies
  • 6 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 6 kudos

Thank you for sharing

  • 6 kudos
mickniz
by Contributor
  • 2840 Views
  • 1 replies
  • 1 kudos

ErrorClass=DAC_DOES_NOT_EXIST]

While creating external table in unity enabled catalog i am geting below error:Data access configuration for metastore does not exist. I can see data access is there.Can anyone let me know if I am misisng here anything

  • 2840 Views
  • 1 replies
  • 1 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 1 kudos

could you share the full error stack trace? how do you create the table? please provide more details, so we can help you to find a solution

  • 1 kudos
Labels