cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
magnus778
by • New Contributor III
  • 2028 Views
  • 2 replies
  • 4 kudos

Resolved! Error writing parquet to specific container in Azure Data Lake

I'm retrieving two files from container1, transforming them and merging before writing to a container2 within the same Storage Account in Azure. I'm mounting container1, unmouting and mounting countainer2 before writing. My code for writing the parqu...

Data Engineering
Reply
  • 2028 Views
  • 2 replies
  • 4 kudos
View Replies
Latest Reply
Pat
Honored Contributor III
  • 4 kudos

Hi @Magnus Asperud​ ,1 mounting container12 you should persist the data somewhere, creating df doesnt mean that you are reading data from container and have it accessible after unmounting. Make sure to store this merged data somewhere. Not sure if th...

  • 4 kudos
1 More Replies
Constantino
by • New Contributor III
  • 2182 Views
  • 3 replies
  • 4 kudos

Resolved! cannot list all tokens with account admin

I'm trying to list all tokens (both user and service principal) for a given workspace; using an Account level admin I've tried both the CLI as well as the API endpoint to list tokens, however each time, only the admin's tokens are returned.I've confi...

Data Engineering
Reply
  • 2182 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
Pat
Honored Contributor III
  • 4 kudos

Great that I could help

  • 4 kudos
2 More Replies
ckwan48
by • New Contributor III
  • 3912 Views
  • 2 replies
  • 4 kudos

Date schema issues with pyspark dataframe creation

I'm having some issues with creating a dataframe with a date column. Could I know what is wrong?from pyspark.sql import SparkSession from pyspark.sql.types import StructType from pyspark.sql.types import DateType, FloatType spark = SparkSession.bui...

Data Engineering
Reply
  • 3912 Views
  • 2 replies
  • 4 kudos
View Replies
Latest Reply
ckwan48
New Contributor III
  • 4 kudos

Hi @Kaniz Fatma​,I actually changed the date format to 'M/d/Y' and it didn't throw any errors. I found in my csv file that it had dates like '3/1/2022'. Could that be the issue? But some dates also were like '12/1/2022. So I'm kind of confused.

  • 4 kudos
1 More Replies
Thanapat_S
by • Contributor
  • 21999 Views
  • 8 replies
  • 5 kudos

Resolved! Can I change from default showing first 1,000 to return all records when query?

I have to query a data for showing in my dashboard.But it truncated the results and showing only first 1,000 rows.In the dashboard view, there is no option to re-execute with maximum result limits.I don't want to switch back to standard view and clic...

image image.png
Data Engineering
Reply
  • 21999 Views
  • 8 replies
  • 5 kudos
View Replies
Latest Reply
Srihasa_Akepati
Databricks Employee
  • 5 kudos

@Thanapat Sontayasara​ 10000 limit is available as an option in the notebook(which propagates to dashboard after its run in notebook) while 1000 rows still is the default. 10000 limit is experimental and it can be made as default depending on the num...

  • 5 kudos
7 More Replies
Saikrishna2
by • New Contributor III
  • 810 Views
  • 0 replies
  • 2 kudos

Databricks SQL user has limitation with 10 queries ?

•Power BI is a publisher that uses AD group authentication to publish result sets. Since the publisher's credentials are maintained, the same user can access the data bricks database.•Number of the users are retrieving the data from the power bi or i...

Data Engineering
Reply
  • 810 Views
  • 0 replies
  • 2 kudos
NSRBX
by • Contributor
  • 3391 Views
  • 8 replies
  • 19 kudos

Databricks-connect not available on Databricks Runtime > 10.4

Hello Databricks Team,Databricks-connect doesn't work on databricks runtime 11.3.Databricks recommends that we use dbx for Databricks Lab instead of databricks-connect. Databricks plans no new feature development for Databricks Connect at this time.D...

Data Engineering
Reply
  • 3391 Views
  • 8 replies
  • 19 kudos
View Replies
Latest Reply
xiangzhu
Contributor III
  • 19 kudos

thx @Landan George​ do you have any ETA about the public preview ?

  • 19 kudos
7 More Replies
AnubhavG
by • Contributor
  • 4982 Views
  • 8 replies
  • 29 kudos

Resolved! Are Python External UDFs supported in Databricks SQL warehouse?

I tried running a python UDF in the Databricks SQL warehouse but it did not run and gave the "Python UDF is not supported" error.Can i get a clear picture if the Python External UDFs are supported or not?

Data Engineering
Reply
  • 4982 Views
  • 8 replies
  • 29 kudos
View Replies
Latest Reply
youssefmrini
Databricks Employee
  • 29 kudos

It's a private preview and It will be supported only in PRO SQL Warehouse and the Serverless SQL Warehouse

  • 29 kudos
7 More Replies
mr_poola49
by • New Contributor III
  • 1871 Views
  • 0 replies
  • 5 kudos

Azure Databricks Jobs Connection Timeout (Read Failed)

Azure Databricks Jobs failed intermittently due to connection timeout (Read Failed) while executing a MS SQL stored procedure which is in Azure SQL database.My requirement is to process delta records(Get delta records using last refresh date) from Da...

Data Engineering
Reply
  • 1871 Views
  • 0 replies
  • 5 kudos
640913
by • New Contributor III
  • 2667 Views
  • 3 replies
  • 2 kudos

Resolved! Databricks hiding error stack trace when plotting?

Hi,I have been working with data visualisations in Databricks the last week and came across a very peculiar behaviour where it seems like Databricks is actively suppressing errors to the user. An example:```import matplotlib.pyplot as pltx = [1, 2, 3...

Data Engineering
Reply
  • 2667 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
LandanG
Databricks Employee
  • 2 kudos

It might be the DBR version. I ran that code on DBR 11.3 and I got the plot + error. Let me confirm if it's the DBR and get back to you here

  • 2 kudos
2 More Replies
Odoo_ERP
by • New Contributor II
  • 2478 Views
  • 0 replies
  • 1 kudos

Odoo ERP customization Odoo is one of the most popular ERP software. It is widely use by companies. Odoo customization mainly includes changing the sy...

Odoo ERP customizationOdoo is one of the most popular ERP software. It is widely use by companies. Odoo customization mainly includes changing the system by including new features and functionalities in accordance with the business needs of the clien...

Data Engineering
Reply
  • 2478 Views
  • 0 replies
  • 1 kudos
Pat
by • Honored Contributor III
  • 1126 Views
  • 0 replies
  • 14 kudos

Automating DDLs and Privilege management

How do you automate table creation and management of privilege / grants on securable objects (tables, views, etc.).I had idea to use terraform to manage above, but terraform doesn't work with managed tables - it allows to create table but then there ...

Data Engineering
Reply
  • 1126 Views
  • 0 replies
  • 14 kudos
elgeo
by • Valued Contributor II
  • 1544 Views
  • 1 replies
  • 2 kudos

Databricks - DBeaver error

Hello experts. While trying to setup the connection between DBeaver and Databricks, we receive the following error:[Databricks][DatabricksJDBCDriver](700120) Host adb-xxxxxxxxxx.azuredatabricks.net cannot be resolved through DnsResolver com.databrick...

Data Engineering
Reply
  • 1544 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
elgeo
Valued Contributor II
  • 2 kudos

We identified the problem. It was due to the proxy that the hostname couldn't be resolved.

  • 2 kudos
Anonymous
by • Not applicable
  • 11114 Views
  • 4 replies
  • 17 kudos

Resolved! SQL query takes too long to run

Hi all,I have a set up SQL query run with 5 hours but the SQL endpoint take too long to start up with each run. Currently I don't know how to fix this Could you please help me how to improve this?

Data Engineering
Reply
  • 11114 Views
  • 4 replies
  • 17 kudos
View Replies
Latest Reply
Unforgiven
Valued Contributor III
  • 17 kudos

or refer on this : Faster SQL Queries on Delta Lake with Dynamic File Pruninghttps://www.databricks.com/blog/2020/04/30/faster-sql-queries-on-delta-lake-with-dynamic-file-pruning.html

  • 17 kudos
3 More Replies
lawrence009
by • Contributor
  • 2919 Views
  • 3 replies
  • 7 kudos

Photon does not fully support the query because of dynamic pruning

Does it still make sense to run this job on a cluster with Photon enable when I am receiving the following?This is the code I ran:CREATE OR REPLACE TABLE ${tbl_name}_dups SELECT src.*, ROW_NUMBER() OVER ( PARTITION BY src.id ...

Data Engineering
Reply
  • 2919 Views
  • 3 replies
  • 7 kudos
View Replies
Latest Reply
PriyaAnanthram
Contributor III
  • 7 kudos

hmm could you show us what your query is

  • 7 kudos
2 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels