cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

HariharaSam
by Contributor
  • 1510 Views
  • 3 replies
  • 0 kudos

DRIVER Garbage Collection

Does anyone know how to fix this ..??

image
  • 1510 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Hariharan Sambath​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....

  • 0 kudos
2 More Replies
KrishZ
by Contributor
  • 1739 Views
  • 4 replies
  • 0 kudos

How to prevent sql queries in 2 notebooks from reading the same row from a Table ?

I have an SQL query to select and update rows in a table. I do this in batches of 300 rows (select 300 , update the selected 300 , select new 300 and update the newly selected and so on..) I run this query in 2 different notebooks concurrently to spe...

  • 1739 Views
  • 4 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Krishna Zanwar​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 0 kudos
3 More Replies
RohitKulkarni
by Contributor II
  • 2125 Views
  • 2 replies
  • 2 kudos

Get file from SharePoint to copy into Azure blob storage

Hello Team,I am trying to copy the xlx files from sharepoint and move to the Azure blob storageUSERNAME = app_config_client.get_configuration_setting(key='BIAppConfig:SharepointUsername',label='BIApp').valuePASSWORD = app_config_client.get_configura...

  • 2125 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Rohit Kulkarni​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 2 kudos
1 More Replies
shanakap
by New Contributor II
  • 1594 Views
  • 2 replies
  • 0 kudos

Create H3 hexagons Mosaic

I am new to using `Mosaic` on Databricks. The task is to create a Heatmap with the counts within Hexagons of H3 resolution. Since the dataset is quite large I am opting for Mosaic. As the first step I am trying to create the hexagons covering the giv...

2ATLt
  • 1594 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Shanaka Pererea​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...

  • 0 kudos
1 More Replies
Ryan_Chynoweth
by Honored Contributor III
  • 1200 Views
  • 3 replies
  • 2 kudos

Hey all, cross posting a medium article I wrote on writing DLT so it dynamically generates tables with less code. Let me know what you think!

Hey all, cross posting a medium article I wrote on writing DLT so it dynamically generates tables with less code. Let me know what you think!

  • 1200 Views
  • 3 replies
  • 2 kudos
Latest Reply
jose_gonzalez
Moderator
  • 2 kudos

Great article, thank you for sharing

  • 2 kudos
2 More Replies
KrishZ
by Contributor
  • 1183 Views
  • 2 replies
  • 1 kudos

Rowlock in databricks sql?

I am trying to get the commands for xlock, holdlock, rowlock, readpast in databricks sql.Looking to implement this

  • 1183 Views
  • 2 replies
  • 1 kudos
Latest Reply
jose_gonzalez
Moderator
  • 1 kudos

Hi @Krishna Zanwar​,We dont currently support these features, but you can create a feature request if you would like to have these features added to Databricks SQL. Please follow the instructions from here https://docs.databricks.com/resources/ideas....

  • 1 kudos
1 More Replies
antoniok
by New Contributor II
  • 2391 Views
  • 1 replies
  • 3 kudos

dbutils.fs.ls is giving "null uri host This can be caused by unencoded / in the password string"

I'm trying to list number of files in s3 bucket. I've initially used "aws s3 ls <s3://>" to list the files and it worked. However, when trying to do the same using dbutils.fs.ls, I'm getting java.lang.NullPointerException: null uri host. This can be ...

  • 2391 Views
  • 1 replies
  • 3 kudos
Latest Reply
marcus1
New Contributor III
  • 3 kudos

You might be encountering an issue with bucket naming. Which I'm also getting with a bucket named something.[0-9]https://issues.apache.org/jira/browse/HADOOP-17241

  • 3 kudos
fhte
by New Contributor
  • 1310 Views
  • 2 replies
  • 0 kudos

How to install R GeoLift library on Databrickts

Hi, I am having problems installing the GeoLift library. I am proceeding as per the official instructions: https://facebookincubator.github.io/GeoLift/docs/GettingStarted/InstallingRThis is what I run in the notebook:1) I install this particular vers...

Screenshot 2022-09-14 at 08.59.09
  • 1310 Views
  • 2 replies
  • 0 kudos
Latest Reply
jose_gonzalez
Moderator
  • 0 kudos

Hi @Ludmila Kuncarova​,I would like to share the following link to our docs https://docs.databricks.com/libraries/notebooks-r-libraries.html in this link you will be able to find more details on how to install R libraries.

  • 0 kudos
1 More Replies
Yuliya
by New Contributor II
  • 1504 Views
  • 2 replies
  • 3 kudos

Azure Databricks SQL Warehouse connection issue

When trying to start SQL Warehouse from my Azure pay-as-you-go subscription, I'm getting error about not enough vCPUs provisioned. Documentation says to increase quota at Azure portal - but it requires knowing type of vCPUs to provision. What type of...

  • 1504 Views
  • 2 replies
  • 3 kudos
Latest Reply
jose_gonzalez
Moderator
  • 3 kudos

Hi @Yuliya Quintela​,Just a friendly follow-up. Did Rostislaw's response help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.

  • 3 kudos
1 More Replies
Frank
by New Contributor III
  • 10038 Views
  • 9 replies
  • 2 kudos

SQLAlchemy ORM Connection String Error

We tried to insert records to Delta table using ORM. It looks like only SQLAlchemy has option to connect to Delta table.We tried the following codefrom sqlalchemy import Column, String, DateTime, Integer, create_engine   engine = create_engine("data...

  • 10038 Views
  • 9 replies
  • 2 kudos
Latest Reply
Ryan_Chynoweth
Honored Contributor III
  • 2 kudos

Hi @Frank Zhang​ , Please disregard the driver comment. The Python SQL Connector requires no driver. Just a pip install and you are good to go. The links you provided don't actually show a working example of using SQL Alchemy's ORM to connect to Data...

  • 2 kudos
8 More Replies
KrishZ
by Contributor
  • 924 Views
  • 2 replies
  • 0 kudos

Where to report a bug with Databricks ?

I have in issue in Pyspark.Pandas to report. Is there a github or some forum where I can register my issue?Here's the issue

  • 924 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Krishna Zanwar​ Does @Debayan Mukherjee​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 0 kudos
1 More Replies
LearningDatabri
by Contributor II
  • 5563 Views
  • 8 replies
  • 9 kudos

repos issue

Why repos works on one workspace and doesn't on another workspace? both have repos enabled.

  • 5563 Views
  • 8 replies
  • 9 kudos
Latest Reply
Prabakar
Esteemed Contributor III
  • 9 kudos

Do you see any errors or what is the issue that you are facing? Could you please describe more about this problem?

  • 9 kudos
7 More Replies
Monika8991
by New Contributor II
  • 2117 Views
  • 2 replies
  • 1 kudos

Getting spark/scala versioning issues while running the spark jobs through Jar

 We tried moving our scala script from standalone cluster to databricks platform. Our script is compatible with following version:Spark: 2.4.8 Scala: 2.11.12The databricks cluster has spark/scala following with version:Spark: 3.2.1. Scala: 2.121: we ...

  • 2117 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Monika Samant​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...

  • 1 kudos
1 More Replies
j_afanador
by Contributor II
  • 1471 Views
  • 1 replies
  • 2 kudos

Resolved! Badge not received for Databricks Lakehouse Fundamentals Accreditation

Hello!I cleared the assessment for Databricks Lakehouse Fundamentals Accreditationbut not received a badge. Kindly assist me with this

  • 1471 Views
  • 1 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Juan Afanador​ Thank you for reaching out! Please submit a ticket to our Training Team here: https://help.databricks.com/s/contact-us?ReqType=training  and our team will get back to you shortly.

  • 2 kudos
Maho
by New Contributor
  • 989 Views
  • 1 replies
  • 1 kudos

Resolved! Lakehouse Fundamentals badge not received

Hi I have finished Lakehouse Fundamentals assessment, received my completion certificate but so far did not receive a badge for it. Would you be able to assist please?

  • 989 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Maciej Oleksy​ Thank you for reaching out! Please submit a ticket to our Training Team here: https://help.databricks.com/s/contact-us?ReqType=training  and our team will get back to you shortly. 

  • 1 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels