cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

GlennStrycker
by • Databricks Partner
  • 6086 Views
  • 1 replies
  • 1 kudos

Resolved! Saving ipynb notebooks to git does not include output cells -- differs from export

When I save+commit+push my .ipynb file to my linked git repo, I noticed that only the cell inputs are saved, not the output.  This differs from the .ipynb file I get when I choose "File / Export / iPython Notebook".  Is there a way to save the cell o...

  • 6086 Views
  • 1 replies
  • 1 kudos
Latest Reply
GlennStrycker
Databricks Partner
  • 1 kudos

I may have figured this out.  You need to allow output in the settings, which will add a .databricks file to your repo, then you'll need to edit the options on your notebook and/or edit the .databricks file to allow all outputs.

  • 1 kudos
YS1
by • Contributor
  • 4831 Views
  • 1 replies
  • 0 kudos

ModuleNotFoundError: No module named 'pulp'

Hello,I'm encountering an issue while running a notebook that utilizes the Pulp library. The library is installed in the first cell of the notebook. Occasionally, I encounter the following error:  org.apache.spark.SparkException: Job aborted due to s...

Data Engineering
Data_Engineering
module_not_found
  • 4831 Views
  • 1 replies
  • 0 kudos
Martinitus
by • Databricks Partner
  • 1873 Views
  • 1 replies
  • 0 kudos

AnalysisException: [ROW_LEVEL_SECURITY_FEATURE_NOT_SUPPORTED.CHECK_CONSTRAINT]

I just tried to setup a row filter via the following two sql snippets:create function if not exists foo.my_test.row_filter (batch_id bigint) return TRUE;alter table foo.my_test.some_table set row filter foo.my_test.row_filter on (batch_id); This resu...

  • 1873 Views
  • 1 replies
  • 0 kudos
Latest Reply
Martinitus
Databricks Partner
  • 0 kudos

 To be fair, row filters and the check constraints feature are in Public Preview, so I apologize for the slightly harsh words above!

  • 0 kudos
Cblunck
by • New Contributor II
  • 6697 Views
  • 3 replies
  • 0 kudos

New to databricks SQL - where clause issue

Hello community,Using Databricks SQL for the first time and I was hoping I could just copy and past my queries from SSMS across and update the table names, but it's not working.Found it's the where statement, which I updated the ' ' to " " but still ...

image.png
  • 6697 Views
  • 3 replies
  • 0 kudos
Latest Reply
justinghavami
New Contributor II
  • 0 kudos

Hi, were you able to get this figured out? I am having the same issue.

  • 0 kudos
2 More Replies
Martinitus
by • Databricks Partner
  • 4884 Views
  • 4 replies
  • 1 kudos

reading a tab separated CSV quietly drops empty rows

I already reported that as a Bug to the official Spark bug tracker: https://issues.apache.org/jira/browse/SPARK-46876A short summary: When reading a tab separated file, that has lines that only contain of tabs, then this line will not show up in the ...

  • 4884 Views
  • 4 replies
  • 1 kudos
Latest Reply
Martinitus
Databricks Partner
  • 1 kudos

@Lakshay Do you know any way to speed up the github merge/review process? The issue has a proposed fix since more than 4 weeks now, but no one seems to care...

  • 1 kudos
3 More Replies
Neha_Gupta
by • New Contributor II
  • 2768 Views
  • 1 replies
  • 0 kudos

Job Concurrency Queue not working as expected

Hi, We have created a Databricks Jobs in Workflows where concurrent runs are set to 10 and the queue is enabled. We were trying to perform concurrent users testing by triggering 100 job runs using Jmeter script. We have observed that the first 10 job...

  • 2768 Views
  • 1 replies
  • 0 kudos
_databreaks
by • Databricks Partner
  • 1516 Views
  • 0 replies
  • 0 kudos

Autolodaer schemaHints convert valid values to null

I am ingesting json files from S3 using Autoloader and would like to use schemaHints to define the datatype of one of the fields, that is, I wanted the field id to be of integer type.The DLT code below infers the the id as string, with correct values...

  • 1516 Views
  • 0 replies
  • 0 kudos
PassionateDBD
by • Databricks Partner
  • 3283 Views
  • 0 replies
  • 0 kudos

MLOps + DLT

What are the best practices for using MLOps and DLT together?This page https://learn.microsoft.com/en-us/azure/databricks/compute/access-mode-limitations states that You cannot use a single user cluster to query tables created by a Unity Catalog-enab...

  • 3283 Views
  • 0 replies
  • 0 kudos
slothPetete
by • New Contributor II
  • 3167 Views
  • 1 replies
  • 0 kudos

Error with mosaic.enable_mosaic() when created DLT Pipeline with Mosaic lib

The error was raised when I tried to start a DLT pipeline with simple code, just to start experimenting the DLT. The primary library was Mosaic, which is already instructed to installed first before importing. The code is roughly as follow $ %pip ins...

Data Engineering
Delta Live Table
dlt
geospatial
mosaic
  • 3167 Views
  • 1 replies
  • 0 kudos
Jaynab_1
by • New Contributor
  • 1847 Views
  • 0 replies
  • 0 kudos

Trying calculate Zonal_stats using mosaic and H3

I am trying to calculate Zonal_stats for raster data using mosaic and H3. Created dataframe from geometry data to H3 index. While previously I was calculating Zonal_stats using rasterio, tif file, geometry data in python which is slow. Now want to ex...

  • 1847 Views
  • 0 replies
  • 0 kudos
ac0
by • Contributor
  • 5539 Views
  • 3 replies
  • 4 kudos

Resolved! Can I have additional logic in a DLT notebook that is unrelated to directly creating DLTs?

I have an Azure Storage Data Table that I would like to update based on records that were just streamed into a Delta Live Table. Below is example code:@Dlt.create_table( comment="comment", table_properties={ "pipelines.autoOptimize.managed": ...

  • 5539 Views
  • 3 replies
  • 4 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 4 kudos

Hi @ac0, Please check @raphaelblg 's response and let us know if this helped to resolve your issue. If it did, please mark it as the accepted solution.

  • 4 kudos
2 More Replies
Jennifer
by • New Contributor III
  • 9775 Views
  • 5 replies
  • 1 kudos

Resolved! Import python file to notebook doesn't work

I followed the documentation here under the section "Import a file into a notebook" to import a shared python file among notebooks used by delta live table. But it sometimes can find the module, sometimes not and returns me exception No module named ...

  • 9775 Views
  • 5 replies
  • 1 kudos
Latest Reply
Vartika
Databricks Employee
  • 1 kudos

Thank you so much for getting back to us @Jennifer MJ​ . It's really great of you to send in the solution. Would you be happy to mark the answer as best so other community members can find the solution quickly and easily? We really appreciate your ti...

  • 1 kudos
4 More Replies
117074
by • New Contributor III
  • 9689 Views
  • 2 replies
  • 0 kudos

Notebook Visualisations suddenly not working

Hi all,I have a python script which runs SQL code against our Delta Live Tables and returns a pandas dataframe. I do this multiple times and then use 'display(pandas_dataframe)'. Once this displays I then create a visualization from the UI which is t...

  • 9689 Views
  • 2 replies
  • 0 kudos
Latest Reply
117074
New Contributor III
  • 0 kudos

Thank you for the detailed response Kaniz, I appreciate it! I do think it may have been cache issues due to there being no spark computation when running them when the error occured.It did lead me down a train of thought.. is it possible to extract t...

  • 0 kudos
1 More Replies
Labels