cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Swostiman
by New Contributor II
  • 4272 Views
  • 5 replies
  • 4 kudos

Consuming data from databricks[Hive metastore] sql endpoint using pyspark

I was trying to read some delta data from databricks[Hive metastore] sql endpoint using pyspark, but while doing so I encountered that all the values of the table after fetching are same as the column name.Even when I try to just show the data it giv...

  • 4272 Views
  • 5 replies
  • 4 kudos
Latest Reply
sucan
New Contributor II
  • 4 kudos

Encountered the same issue and downgrading to 2.6.22 helped me resolve this issue.

  • 4 kudos
4 More Replies
pg1
by New Contributor II
  • 889 Views
  • 2 replies
  • 1 kudos

How to expand display of workspace name in top right window?

How to expand display of workspace name in top right window? My workspace name is little bit longer and it cuts out, could not find a way to expand name window.

  • 889 Views
  • 2 replies
  • 1 kudos
Latest Reply
Tharun-Kumar
Honored Contributor II
  • 1 kudos

@pg1 Is your workspace Unity Catalog enabled?We have identified this behaviour in Unity Catalog enabled workspaces and already working to show up the entire workspace name when we hover over it.

  • 1 kudos
1 More Replies
cbabay
by New Contributor II
  • 1428 Views
  • 3 replies
  • 2 kudos

Resolved! Adding Duration Warning During Workflow Deployment

Hi Team,Thank you very much for making it so simple to add notifications on failure and duration warnings to workflows.Each time we deploy to dev, through Jenkins, our email is unsubscribed and we stop getting alerts. Is there any documentation on ho...

  • 1428 Views
  • 3 replies
  • 2 kudos
Latest Reply
alexott
Valued Contributor II
  • 2 kudos

It heavily depends on how do you perform deployment - what APIs are used, etc. Most probably you're using Job Reset API instead of Job Update API

  • 2 kudos
2 More Replies
Yasser
by New Contributor
  • 567 Views
  • 0 replies
  • 0 kudos

[sql warehouse] Invalid configuration value detected for fs.azure.account.key with 'force' = 'true'

Hello,I am getting the following error when trying to copy data to databricks from an ADLS with SQL and using a SAS tokenFailure to initialize configuration for storage account <storage account>: Invalid configuration value detected for fs.azure.acco...

  • 567 Views
  • 0 replies
  • 0 kudos
Chris_Shehu
by Valued Contributor III
  • 12322 Views
  • 5 replies
  • 5 kudos

Resolved! What is the best way to handle big data sets?

I'm trying to find the best strategy for handling big data sets. In this case I have something that is 450 million records. I'm pulling the data from SQL Server very quickly but when I try to push the data to the Delta Table OR a Azure Container the...

  • 12322 Views
  • 5 replies
  • 5 kudos
Latest Reply
Wilynan
New Contributor II
  • 5 kudos

I think you should consult experts in Big Data for advice on this issue

  • 5 kudos
4 More Replies
parthsalvi
by Contributor
  • 1907 Views
  • 1 replies
  • 2 kudos

Amazon SES : boto3 credentials not found. DBR 11.2 Shared mode

We're trying to send email using Amazon SES using boto3.client in python. We've added SES Full access in clusters IAM Role.We were able to send email in "No isolation shared" mode in DBR 11.2 using ses = boto3.client('ses', region_name='us-****-2') s...

image
  • 1907 Views
  • 1 replies
  • 2 kudos
Latest Reply
JameDavi_51481
New Contributor III
  • 2 kudos

This appears to be an intentional design choice to prevent users from using the credentials of the host machine to carry out arbitrary AWS API calls. I really wish there was a workaround or setting to disable this behavior because we put a lot of wor...

  • 2 kudos
DatabricksHero
by New Contributor II
  • 1571 Views
  • 2 replies
  • 0 kudos

Unity Catalog 2.1 API Not Returning SQL Function/View Dependencies

Hi all,I have a problem with reading responses generated by Unity Catalog API 2.1 as they are missing fields that are otherwise described in the specification:List functions - The fields routine_dependencies, return_params, and input_params are missi...

Data Engineering
API
sql
Unity Catalog
  • 1571 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @DatabricksHero ,    view_dependencies View dependencies (when table_type == VIEW or MATERIALIZED_VIEW, STREAMING_TABLE) when DependencyList is None, the dependency is not provided;when DependencyList is an empty list, the dependency is provided b...

  • 0 kudos
1 More Replies
Henrik
by New Contributor III
  • 1077 Views
  • 1 replies
  • 1 kudos

Resolved! Run notebooks on serverless SQL cluster

Is it just me or i'm I right that we  can't run notebooks on a serverless SQL cluster?It would be a nice feature for SQL based notebooks.

  • 1077 Views
  • 1 replies
  • 1 kudos
Latest Reply
Henrik
New Contributor III
  • 1 kudos

I figured out.I needed to start the cluster first.

  • 1 kudos
Sinthiya
by New Contributor II
  • 1544 Views
  • 1 replies
  • 1 kudos

Multiple streaming sources to the single delta live table

In our case, we have multiple sources writing to the same target table.  A target table can be populated from multiple source tables, each contributing a set of fields. How to add/update columns in a target table from multiple sources.In a delta live...

  • 1544 Views
  • 1 replies
  • 1 kudos
Latest Reply
SaiKiranGajjala
New Contributor II
  • 1 kudos

Following.

  • 1 kudos
a_t_h_i
by New Contributor
  • 1982 Views
  • 1 replies
  • 1 kudos

Move managed DLT table from one schema to another schema in Databricks

I have a DLT table in schema A which is being loaded by DLT pipeline.I want to move the table from schema A to schema B, and repoint my existing DLT pipeline to table in schema B. also I need to avoid full reload in DLT pipeline on table in Schema B....

Data Engineering
delta-live-table
deltalivetable
deltatable
dlt
  • 1982 Views
  • 1 replies
  • 1 kudos
Latest Reply
Tharun-Kumar
Honored Contributor II
  • 1 kudos

@a_t_h_i This feature is being actively worked upon by our Engineers. The plan is to change the schema name in the DLT pipeline settings and DLT will move the managed DLT table to the other schema.

  • 1 kudos
Mr_K
by New Contributor
  • 4395 Views
  • 2 replies
  • 2 kudos

AnalysisException: [UC_COMMAND_NOT_SUPPORTED] Spark higher-order functions are not supported in Unity Catalog.;

Hello,forecast_date = '2017-12-01' spark.conf.set('spark.sql.shuffle.partitions', 500 ) # generate forecast for this data forecasts = ( history .where(history.date < forecast_date) # limit training data to prior to our forecast date .groupBy...

  • 4395 Views
  • 2 replies
  • 2 kudos
Latest Reply
Tharun-Kumar
Honored Contributor II
  • 2 kudos

@Mr_K ApplyInPandas is a higher order function in Python. As of now, we do not support higher order functions in Unity Catalog. We do support direct calls made to python UDFs. Here is an example of how to reference UDFs in UC - https://docs.databrick...

  • 2 kudos
1 More Replies
schnee1
by New Contributor III
  • 7211 Views
  • 8 replies
  • 0 kudos

Access struct elements inside dataframe?

I have JSON data set that contains a price in a string like "USD 5.00". I'd like to convert the numeric portion to a Double to use in an MLLIB LabeledPoint, and have managed to split the price string into an array of string. The below creates a data...

  • 7211 Views
  • 8 replies
  • 0 kudos
Latest Reply
goldentriangle
New Contributor II
  • 0 kudos

Thanks, Golden Triangle Tour

  • 0 kudos
7 More Replies
NCat
by New Contributor III
  • 4770 Views
  • 5 replies
  • 0 kudos

ipywidgets: Uncaught RefferenceError require is not defined

Hi,When I tried to use ipywidgets, it returns the following error.I’m using Databricks with PrivateLink enabled on AWS, and Runtime version is 12.2 LTS.Is there something that I need to use ipywidgets in my environment?

CA0045C4-83C6-46FC-95DC-6857199FE69D.jpeg
  • 4770 Views
  • 5 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @NCat, The error message "uncaught reference error: require is not defined" indicates that the require function is not defined in the current scope. This error can occur when using Databricks Connect with a version of Node.js that does not support...

  • 0 kudos
4 More Replies
MattM
by New Contributor III
  • 4983 Views
  • 8 replies
  • 2 kudos

Resolved! Access Databricks Delta table using SSRS without copying data to AzureSQL

We have our BI facts and dimensions built in as delta table in Datarbicks env and is being used for reporting by connecting PowerBI reports using datarbricks connection. We now have a need to use this data for another application utilizing SSRS repor...

  • 4983 Views
  • 8 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

https://buyusasmm.com/product/buy-google-5-star-reviews/

  • 2 kudos
7 More Replies
kll
by New Contributor III
  • 12103 Views
  • 3 replies
  • 0 kudos

python multiprocessing and the Databricks Architecture - under the hood.

I am curious what is going on under-the-hood when using `multiprocessing` module to parallelize an function call and apply it to a Pandas DataFrame along the row axis. Specifically, how does it work with DataBricks Architecture / Compute. My cluster ...

  • 12103 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

@Keval Shah​ :When using the multiprocessing module in Python to parallelize a function call and apply it to a Pandas DataFrame along the row axis, the following happens under the hood:The Pool object is created with the specified number of processes...

  • 0 kudos
2 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels