cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
raj_123469
by • New Contributor II
  • 1431 Views
  • 2 replies
  • 2 kudos

how to access the jupyter notebook

Data Engineering
Reply
  • 1431 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 2 kudos

Hi @rajat kumar​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...

  • 2 kudos
1 More Replies
niels
by • New Contributor III
  • 1635 Views
  • 2 replies
  • 0 kudos

Azure SA mounted but can't load files

I am attempting to load an excel file that's located in a blob storage that I've mounted. In the first cell, when I use the dbutils.fs.ls command, I can see the file I want to load. However, when I try to actually load it, it can't find the file. It ...

Data Engineering
Reply
  • 1635 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hi @Niels Ota​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 0 kudos
1 More Replies
Herkimer
by • New Contributor II
  • 1841 Views
  • 3 replies
  • 0 kudos

Is it possible to install databricks-cli on a shared laptop.

I have a government furnished laptop (GFE). My normal user is not an admin on the laptop. I have a separate admin login on the laptop. I was able to install databricks-cli as the admin user but it installed under that users appdata python path which ...

Data Engineering
Reply
  • 1841 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hey there @John Zajic​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....

  • 0 kudos
2 More Replies
RantoB
by • Valued Contributor
  • 4998 Views
  • 3 replies
  • 4 kudos

Resolved! DeltaTable' object has no attribute 'clone'

Hello, I use delta on my local machine and I would like to clone a table, however the cloning is not working.I have the last version of delta installed (delta-spark==2.0.0) but the clone method does not exist in the python module.With this code :delt...

Data Engineering
Reply
  • 4998 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 4 kudos

Hello @Bertrand BURCKER​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from yo...

  • 4 kudos
2 More Replies
Tahseen0354
by • Valued Contributor
  • 1539 Views
  • 2 replies
  • 0 kudos

User's name is empty sometimes when a new user is added from admin console

Hi, when I add a new user from admin console, the name of the user is empty. It does not happen all the time. For some users, the username and name both are available. But for some new users, the value in the name column in the users list is empty. W...

Data Engineering
Reply
  • 1539 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Hi, when I add a new user from admin console, the name of the user is empty. It does not happen all the time. For some users, the username and name both are available. But for some new users, the value in the name column in the users list is empty. W...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
1 More Replies
Chris_Shehu
by • Valued Contributor III
  • 5924 Views
  • 8 replies
  • 2 kudos

Resolved! Do compute resources get removed after not being used for x number of days?

Currently we're getting reports of compute resources disappearing from one of our lesser used databricks platforms. I just turned on logging to see if we can find something but I'm wondering if a compute gets removed if it hasn't been used after so l...

Data Engineering
Reply
  • 5924 Views
  • 8 replies
  • 2 kudos
View Replies
Latest Reply
Hanna0805050
New Contributor II
  • 2 kudos

Pest Control Software to Grow Your Business Choosing the best pest control software for your business can have a powerful impact on your productivity. Fieldwork can help your workforce repel downtime, attract clients, get organized and get everything...

  • 2 kudos
7 More Replies
Data_Engineer3
by • Contributor III
  • 5802 Views
  • 4 replies
  • 1 kudos

Unable to read data from Elasticsearch with spark in Databricks.

When I am trying to read data from elasticsearch by spark sql, it throw an error like RuntimeException: Error while encoding: java.lang.RuntimeException: scala.collection.convert.Wrappers$JListWrapper is not a valid external type for schema of string...

Data Engineering
Reply
  • 5802 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 1 kudos

Hi there @KARTHICK N​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.T...

  • 1 kudos
3 More Replies
rodrigocms
by • New Contributor
  • 1955 Views
  • 2 replies
  • 0 kudos

Connect to SSAS

Hello everyone,I need to connect Databricks Pyspark to get information from Power BI XLMA EndPoint - the end point work as an SSAS host.So, I'm trying to find what I need to do to connect to SSAS tabular. Can anyone help?Many thanks.Rodrigo Souza

Data Engineering
Reply
  • 1955 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hey there @Rodrigo Camara de Souza​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to h...

  • 0 kudos
1 More Replies
GoldenTuna
by • New Contributor II
  • 2426 Views
  • 3 replies
  • 1 kudos

Bulk removal of inactive users?

To make a long story short, through SCIM we accidentally provisioned 3,000+ users into our Databricks workspace who should not be there. We fixed the SCIM issue but now the workspaces tab is flooded with inactive user workspaces. Is there any way to ...

Data Engineering
Reply
  • 2426 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 1 kudos

Hello @David Kruetzkamp​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from yo...

  • 1 kudos
2 More Replies
sabooalex
by • New Contributor II
  • 794 Views
  • 0 replies
  • 0 kudos

SCD type2 snowflake

I have monthly files which comes in S3 bucket. I want to implement SCD type2 in snowflake.I am ok to read the new files, clean it.My question is about comparing what I have read from the files, with what is stored in the snowflake table already(milli...

Data Engineering
Reply
  • 794 Views
  • 0 replies
  • 0 kudos
Anonymous
by • Not applicable
  • 1098 Views
  • 3 replies
  • 0 kudos

The Next Databricks Office HoursOur next Office Hours session is scheduled for January 25, 2022 - 8:00 am PDT Do you have questions about how to set u...

The Next Databricks Office HoursOur next Office Hours session is scheduled for January 25, 2022 - 8:00 am PDTDo you have questions about how to set up or use Databricks? Do you want to get best practices for deploying your use case or tips on data ar...

Data Engineering
Reply
  • 1098 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Hanna0805050
New Contributor II
  • 0 kudos

Thank you for the opportunity to communicate. I work at https://www.eliteimagingsystems.com/ and know how important it is for our customers to be able to communicate with us 24/7.

  • 0 kudos
2 More Replies
jwilliam
by • Contributor
  • 3119 Views
  • 3 replies
  • 4 kudos

Resolved! What is the maximum of concurrent streaming jobs for a cluster?

What is the maximum of concurrent streaming jobs for a cluster? How can I have the right amount of concurrent streaming jobs for different cluster configuration?Should I use multiple cluster for different jobs or combine it into a big cluster to hand...

Data Engineering
Reply
  • 3119 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
Prabakar
Databricks Employee
  • 4 kudos

Hi @John William​ it would be better to use different clusters for each streaming jobs.

  • 4 kudos
2 More Replies
DipakBachhav
by • New Contributor III
  • 10407 Views
  • 5 replies
  • 1 kudos

How to store SQL query result data on a local disk?

I am a newbie to data bricks and trying to write results into the excel/ CSV file using the below command but getting DataFrame' object has no attribute 'to_csv' errors while executing.I am using a notebook to execute my SQL queries and now want to s...

Data Engineering
Reply
  • 10407 Views
  • 5 replies
  • 1 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 1 kudos

Hi there @Dipak Bachhav​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from yo...

  • 1 kudos
4 More Replies
Ruby8376
by • Valued Contributor
  • 3697 Views
  • 8 replies
  • 1 kudos

Resolved! Anti pattern : moving data from cloud to on-prem

Hi there,In my current project, Current status: Az databricks streaming jobs migrate Json file from kafka to raw layer(parquet file), then parsing logic is applied and 8 tables are created in raw standardized layer.Requirement: Business team wants to...

Data Engineering
Reply
  • 3697 Views
  • 8 replies
  • 1 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 1 kudos

You could indeed use ADF to copy the data from cloud to on-prem.However, depending on the size of the data, this can take a while.I use the same pattern, but for aggregated processed data, which is not an issue at all.You could also look at Azure Syn...

  • 1 kudos
7 More Replies
MrT
by • New Contributor II
  • 2717 Views
  • 3 replies
  • 3 kudos

Python databricks-sql-connector TLS issue - client tries to negotiate v1 which fails many times then randomly tries to negotiate v1.3 which works

This issue is oddly only on an Azure Windows 10 VM. I Dont have this on my workstation or my personal computer so it seems to be host config related. The VM where the issue is i have a simple python script that connects to the Azure Databricks SQL en...

image
Data Engineering
Reply
  • 2717 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 3 kudos

Hello @Wayne Theron​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...

  • 3 kudos
2 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels