cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Kyler
by New Contributor
  • 876 Views
  • 2 replies
  • 0 kudos

ILT session

when is the next ILT session

  • 876 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Edrian Kyle​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback ...

  • 0 kudos
1 More Replies
sribet
by New Contributor
  • 1052 Views
  • 2 replies
  • 0 kudos

Adding partition location to external table throwing exception

I have created external table like below.# create table spark.sql(f""" CREATE EXTERNAL TABLE IF NOT EXISTS {database_schema}.{tableName} USING PARQUET OPTIONS ( 'path' '{raw_storage}/{folder_path}', 'forward_spark_azur...

  • 1052 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @sri bet​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback will...

  • 0 kudos
1 More Replies
Dave_Nithio
by Contributor
  • 2220 Views
  • 2 replies
  • 1 kudos

Skip Task Without Spinning Up Cluster

I have a Job Workflow with multiple sequential tasks executing R or Python scripts. Currently, we can skip one of these tasks (if it has already been run) by passing a parameter and skipping via the script. This requires a full spin up of a compute r...

  • 2220 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Dave Wilson​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...

  • 1 kudos
1 More Replies
PFA
by New Contributor
  • 1815 Views
  • 2 replies
  • 0 kudos

Why is the Import button greyed out when attempting to import a .ipynb file into Databricks?

The file import seems to work. The file name and size appear in the dialog box along with a green check mark. BUT only the cancel button is active. The import button is greyed out.Is there another step, action, or setting required to activate the i...

  • 1815 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Paul Auclair​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 0 kudos
1 More Replies
MaheshDR
by New Contributor II
  • 1333 Views
  • 2 replies
  • 0 kudos

Informatica Cloud mapping with Databricks connection failing with java.util.NoSuchElementException

Hi Team,When we tried to configure our source with Databricks table with Databricks connection on Informatica Cloud, we received below error.We already tried the suggestions mentioned in the below community post which seems to be similar error as our...

  • 1333 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Mahesh D​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 0 kudos
1 More Replies
674491
by New Contributor II
  • 1295 Views
  • 2 replies
  • 1 kudos

Is it possible to get a git access token on the databricks notebook.

I set my git access token with folloing the page.Get a Git access token & connect a remote repo to Databricks | Databricks on AWSI would like to get that git token on my databricks python notebook.Is it possible? Or, should I set same or another git ...

  • 1295 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @takeshi.kida​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 1 kudos
1 More Replies
Sumitsn85
by New Contributor
  • 3394 Views
  • 2 replies
  • 0 kudos

Maximum VM usage

Hi, I am trying to find what should be the maximum VM's i should reserve as capacity.I use F8 Cluster instance pool. Multiple jobs use this instance pool during the day.Most of them overlap during different times of the day.For reserving capacity, i ...

  • 3394 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Sumit Salian​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback...

  • 0 kudos
1 More Replies
shelly
by New Contributor
  • 1648 Views
  • 3 replies
  • 0 kudos

take() operation throwing index out of range error

x=[1,2,3,4,5,6,7]rdd = sc.parallelize(x)print (rdd.take(2))Traceback (most recent call last): File "/usr/local/spark/python/pyspark/serializers.py", line 458, in dumps return cloudpickle.dumps(obj, pickle_protocol) ^^^^^^^^^^^^^^^^^^...

  • 1648 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Shelly Bhardwaj​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...

  • 0 kudos
2 More Replies
hitesh22
by New Contributor II
  • 2773 Views
  • 5 replies
  • 0 kudos
  • 2773 Views
  • 5 replies
  • 0 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 0 kudos

Hi, I am not sure if this helps: https://www.databricks.com/blog/2020/12/15/python-autocomplete-improvements-for-databricks-notebooks.htmlAlso, please tag @Debayan​ with your next response which will notify me. Thank you!

  • 0 kudos
4 More Replies
fuselessmatt
by Contributor
  • 2609 Views
  • 3 replies
  • 0 kudos

Omitting columns in an INSERT statement does not seem to work despite meeting the requirements

We want to use the INSERT INTO command with specific columns as specified in the official documentation. The only requirements for this are​️ Databricks SQL warehouse version 2022.35 or higher️ Databricks Runtime 11.2 and above​and the behaviour shou...

  • 2609 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Fusselmanwog​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...

  • 0 kudos
2 More Replies
Kaniz
by Community Manager
  • 1302 Views
  • 3 replies
  • 4 kudos

Databricks SQL is curious !!!!

Hello Databricks Community!We're conducting a quick poll to gather insights on your experience with SQL in Databricks.Your input will help us tailor our content and discussions to serve the community's needs better. Please take a moment to answer the...

  • 1302 Views
  • 3 replies
  • 4 kudos
Latest Reply
rendorHaevyn
New Contributor III
  • 4 kudos

If you're asking about the Databricks SQL service: (8) (8.1) Repos integration of SQL Queries (requested feature)(8.2) Drag-and-drop GUI organisation of SQL queries and dashboards by features / tags / user, and generally a significantly improved UI (...

  • 4 kudos
2 More Replies
MetaRossiVinli
by Contributor
  • 2061 Views
  • 2 replies
  • 4 kudos

Resolved! Can you use the Secrets API 2.0 in a Delta Live Tables configuration?

Is the Secrets API 2.0 not applied to Delta Live Tables configurations? I understand that the Secrets API 2.0 is in public preview and this use case may not be supported, yet. I tried the following and both do not work for the stated reasons.In a DLT...

  • 2061 Views
  • 2 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

@Kevin Rossi​ : As a workaround, you can use the code you provided to load the secret in a cell in a DLT notebook and set it in the Spark configuration. This will allow you to use the secret in your DLT code.Another workaround could be to store the c...

  • 4 kudos
1 More Replies
mshettar
by New Contributor II
  • 1165 Views
  • 2 replies
  • 2 kudos

newAPIHadoopRDD Spark API doesn't retrieve unflushed data written to Hbase table

Reading from an HBase table with a few hundred records that haven't been persisted (flushed) to HDFS doesn't show up in Spark. However, the records become visible after forced flush via Hbase shell or system triggered flush (when size of Memstore cro...

  • 1165 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

@Manjunath Shettar​ :It seems that the issue is related to the fact that the records in the HBase table have not been flushed to HDFS and are still stored in the Memstore. Spark's newAPIHadoopRDD API reads data from the HBase table through HBase's Ta...

  • 2 kudos
1 More Replies
Osky_Rosky
by New Contributor II
  • 3740 Views
  • 2 replies
  • 0 kudos

Combine Python + R in data manipulation in Databricks Notebook

Want to combine Py + Rfrom pyspark.sql import SparkSessionspark = SparkSession.builder.appName("CreateDataFrame").getOrCreate()# Create a sample DataFramedata = [("Alice", 25), ("Bob", 30), ("Charlie", 35), ("Oscar",36), ("Hiromi",41), ("Alejandro", ...

  • 3740 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

@Oscar CENTENO MORA​ :To combine Py and R in a Databricks notebook, you can use the magics command %python and %rto switch between Python and R cells. Here's an example of how to create a Spark DataFrame in Python and then use it in R:from pyspark.sq...

  • 0 kudos
1 More Replies
TheRealJimShady
by New Contributor
  • 4023 Views
  • 7 replies
  • 0 kudos

Resolved! Email destination not appearing in Job's System Notification list.

On job failure I need to send an email with a custom subject line. I have configured the email address as a destination with the subject that I need, but I don't see it as an option that I can choose in the 'System Notification' dialog in the job set...

Screenshot 2023-03-30 161113 Screenshot 2023-03-30 161231
  • 4023 Views
  • 7 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @James Smith​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so w...

  • 0 kudos
6 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels